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Optimal  Structures  for  Multimedia  Instruction: 

Final  Report 


by  Joseph  A.  Goguen,  Charlotte  Linde  and  T.  K.  Bikson 
SRI  International 
333  Ravenswood  Avenue 
Menlo  Park  CA  94025 

1  Introduction 

This  final  report  presents  the  results  of  a  two-year  study  of  the  structure  of  multimedia 
instruction.  The  major  findings  include  the  following: 

•  Choice  of  discourse  type  can  strongly  affect  comprehension;  this  is  a  new  result  that 
would  not  have  been  predicted  by  any  current  theory  of  instruction  or  comprehension. 

•  Gestural  coordination  between  visual  and  verbal  material  can  significantly  help 
comprehension. 

•  Suitable  visual  icons  can  greatly  aid  comprehension,  even  when  they  are  not  coordinated 
with  linguistic  material. 

•  Cognitive  level  has  a  very  significant  and  systematic  effect  on  comprehension. 

•  Manipulation  of  the  internal  structure  of  discourse  units  has  significant  and  systematic 
effects;  in  particular,  the  greater  the  level  of  structure  violated,  the  greater  seems  to  be  its 
effect  on  comprehension.  To  discover  this  effect  requires  a  relatively  fine-grained  theory 
of  discourse  structure. 

•  Inclusion  of  summaries,  particularly  initial  summaries,  seems  to  assist  comprehension. 
These  results  seem  quite  encouraging  for  a  new  research  paradigm  in  such  a  complex  area  as 
multimedia  instruction.  Moreover,  they  suggest  some  interesting  directions  for  further  research. 
They  also  have  specific  implications  for  instructional  design,  as  detailed  in  Sections  5.3  and  5.4, 
respectively. 

1.1  The  Problem 

The  real-world  problem  that  has  inspired  this  research  is  how  to  design  effective  multimedia 
instructional  materials.  There  is,  of  course,  a  vast  body  of  information  on  learning  and  the 
bases  of  effective  learning  (for  example,  see  [Estes  78]).  However,  there  has  been  little  or  no 
research  on  the  effects  of  the  discourse  structure  of  instructional  materials  or  the  coordination 
of  multiple  instructional  media.  Also,  new  technology  has  introduced  new  media,  such  as 
interactive  computer  graphics,  as  well  as  new  possibilities  for  combining  media  in  the 
instructional  situation  (e.g.,  using  videodisc).  (See  [Ong  82,  Ong  81]  for  a  discussion  of  the 
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history  of  profound  changes  imposed  on  educational  systems  by  the  introduction  first  of  writing, 
and  later,  of  printing.) 

For  conventional  instructional  media  there  are,  if  not  general  design  principles,  at  least  a  body 
of  experience  and  folk  wisdom  about  how  to  proceed.  But  there  is  no  such  accumulated 
experience  for  the  production  of  materials  using  video  and  computer  technology.  Furthermore, 
such  media  permit,  and  indeed  almost  require,  much  greater  complexity  of  design  than 
conventional  media.  Therefore,  in  designing  such  materials,  one  should  use  theoretically 
motivated  and  experimentally  verified  principles  for  choosing  among  the  many  possible 
instructional  structures  and  representations  possible  using  the  various  media  involved.  This 
report  presents  research  aimed  at  exploring  some  basic  principles  of  this  kind. 

1.2  Technical  Approach 

This  study  takes  a  multidisciplinary  approach  to  the  problem  of  discovering  principles  for 
designing  multimedia  instruction.  The  disciplines  include  experimental  psychololgy,  discourse 
analysis  (the  study  of  linguistic  units  larger  than  the  sentence)  and  semiotics  (the  study  of  the 
general  principles  of  sign  systems,  such  as  language,  visual  gesture,  diagrams,  and  mixtures  of 
these). 

The  experimental  studies  build  on  our  prior  naturalistic  studies  of  the  structure  of  explanation 
(see  (Goguen  &  Linde  84]).  In  these  prior  studies,  we  devised  an  experimental  task,  built  a 
demonstration  device,  and  elicited  explanations  about  its  operation  from  experienced  instructors 
(engineering  professors  at  a  local  community  college).  These  explanations  were  videotaped  and 
analyzed  to  determine  the  range  of  explanation  structures,  use  of  diagrams,  gestures,  and  the 
like  in  spontaneous  instructional  discourse.  This  body  of  information  was  then  used  in 
generating  the  hypotheses  tested  in  subsequent  phases  of  this  study. 

The  task  domain  used  in  the  experiments  reported  here  was  essentially  the  same  as  for  the 
elicitation  study:  a  logic  box  said  to  control  an  irrigation  system  (see  Figure  5).  Fifteen 
different  stimuli  were  prepared,  each  including  an  explanation  of  how  the  box  works;  each  was 
shown  to  a  group  of  approximately  25  volunteer  subjects,  who  were  local  community  college 
students.  The  stimuli  were  videotapes  of  a  professional  actor  following  a  carefully  prepared 
multimedia  script,  involving  use  of  verbal  explanation,  diagrams  on  a  blackboard,  and  the  box 
itself.  The  stimuli  comprise  two  series,  the  first  experimental  series  employs  a  fully-crossed 
design  with  three  binary  independent  variables;  the  second  explores  seven  variants  of  a  single 
explanation  type. 


The  variables  in  the  first  series  were:  Discourse  Type  (Pseudonarrative  versus  Reasoning), 
presence  or  absence  of  discourse  Structure  Marking;  and  presence  or  absence  of  Visual  Deixis. 
All  explanations  in  this  series  were  at  the  Combinatorial  cognitive  level1;  that  is,  the  lights  were 
directly  keyed  to  switch  patterns,  rather  than  to  the  Boolean  functions  that  generate  those 
relations. 

All  explanations  in  the  second  series  used  reasoning  discourse  structure.  Experimental 
manipulations  were  designed  to  explore  effects  on  comprehension  of:  presence  of  initial 
summary,  presence  of  final  summary,  presence  of  focus  inconsistencies  at  the  sentence  level, 
presence  of  discourse  unit  boundary  violations,  presence  of  discourse  section  scrambling,  and 
visual  icon  not  coordinated  with  verbal  discourse.  All  explanations  in  the  second  series  were  at 
the  Boolean  cognitive  level;  that  is  the  Boolean  relations  between  the  lights  and  switches  were 
explained  (however,  the  uncoordinated  icon  represented  the  combinatorial  level). 

Separate  objective  tests  were  devised  for  each  series,  sharing  several  questions  to  permit  cross¬ 
series  comparisons.  Scores  on  these  questionnaires  were  used  as  the  dependent  measure. 
Analyses  relied  chiefly  on  1-  and  3-way  ANOVAS,  using  t  tests  to  evaluate  between  group 
differences;  in  addition,  correlations  between  individual  items  and  total  test  scores  were  used  to 
determine  the  appropriateness  of  a  summary  dependent  measure.  A  number  of  hypotheses  were 
tested  using  these  procedures,  resulting  in  both  the  conclusions  already  summarized,  and  several 
practical  recommendations. 

2  Theoretical  Background 

This  section  discusses  three  theoretical  systems  underlying  in  this  work:  discourse  analysis, 
cognitive  level,  and  semiotics.  It  then  introduces  some  experimental  variables  suggested  by 
these  theories. 

2.1  Discourse  Type 

In  this  research,  discourse  analysis  is  used  as  a  major  tool  for  the  generation  of  hypotheses, 
production  of  experimental  material,  and  interpretation  of  results.  Discourse  analysis  takes  as 
data  extended  stretches  of  language  occurring  in  a  natural  setting,  in  this  case,  instructional 
discourse.  It  then  determines  the  discourse  units  present,  and  analyzes  their  formal  structure. 
(For  detailed  discussion  of  the  methodological  basis  of  discourse  analysis,  see  [Linde  81,  Linde  & 

"See  Section  2.2  for  further  discussion  of  cognitive  levels. 


Goguen  80];  for  an  example  of  the  practical  application  of  this  method  to  context  of  commercial 
aviation  communication  see  [Goguen  &  Linde  83].) 

A  discourse  unit  is  a  segment  of  spoken  language,  longer  than  a  single  sentence,  with  socially 
recognized  initial  and  final  boundaries,  and  a  formally  definable  internal  structure.  A 
discourse  type  is  a  theory  of  the  structure  of  a  class  of  discourse  units.  Major  discourse  types 
that  have  been  studied  include: 

•  the  narrative  of  personal  experience  [Labov  72), 

•  the  pseudonarrative,  e.g.,  spatial  descriptions  [Linde  74,  Linde  &  Labov  75], 

•  small  group  planning  [Linde  &  Goguen  78], 

•  the  joke  [Sacks  74]  and 

•  reasoning  [Weiner  79,  Goguen,  Weiner  &  Linde  83]. 

Analysis  of  the  taped  naturalistic  explanation  sessions  showed  that  the  major  discourse  units 
used  in  instructional  discourse  are  reasoning,  plans,  and  pseudonarrative.  A  discourse  type 
provides  a  tree  structure  for  its  discourse  units,  breaking  it  down  into  subparts,  which 
themselves  may  be  further  decomposed.  These  tree  structures  can  be  seen  to  result  from  the 
application  of  a  sequence  of  transformations  [Linde  &  Goguen  78,  Goguen,  Weiner  &  Linde 
83,  Goguen  &  Linde  84]  representing  the  real-time  effects  of  additions  and  modifications  of  the 
reasoning  structure,  as  well  as  movements  of  the  focus  of  attention.  Thus  the  sequence  of  trees 
represents  the  individual  or  group  process  of  the  production  of  a  discourse  unit,  and  the  final 
tree  represents  the  finished  unit.  Each  discourse  type  has  distinctive  node  types  which 
characterize  it,  as  well  as  some  node  types  such  as  AND  which  seem  to  be  common  to  all 
discourse  types. 

Discourse  structures  may  be  indicated  in  spontaneous  speech  either  by  explicit  discourse 
structure  markers,  such  as  first,  next,  and  finally  which  indicate  current  position,  or 
implicitly  by  the  semantics  of  the  component  sentences.  Below  is  a  pseudonarrative  stimulus 
passage,  as  used  in  the  current  experiments,  with  its  structure  markers  italicized: 

So  to  begin,  ve  have  this  controller  with  tvo  switches  and  four  lights 
and  we  have  to  learn  how  to  operate  it.  Now  let's  make  sure  that 
there  really  are  just  four  possible  ways  to  set  these  two  switches. 

Ve'll  start  with  the  first  setting,  which  has  both  switches  up.  Then  we 
go  to  the  second  setting,  which  has  both  switches  down.  Then  we  go  to 
the  third  setting,  which  has  the  first  switch  up  and  the  second  switch 
down.  Finally,  we  go  to  the  fourth  setting,  which  has  the  first  switch 
down  and  the  second  switch  up.  And  that’s  all  the  settings  that  there 
are.  So  this  fertilizer  controller  can  only  select  among  four  mixtures 
of  chemicals  to  add  to  the  water. 


Such  discourse  structure  markers  indicate  the  organization  of  the  discourse  unit;  they  mark  the 
beginnings  of  new  subtrees,  and  hence  correspond  to  a  movement  of  focus  of  attention  first  up 
the  structure  tree,  and  then  back  down  to  the  root  of  a  new  subtree.  Thus,  in  the  example 
above,  the  so  in  the  final  sentence  indicates  a  movement  of  the  focus  of  attention  back  to  the 
top  node  of  the  discourse.  Markers  of  this  type  are  called  "POP*  markers. 

The  following  subsections  briefly  discuss  reasoning  and  pseudonarrative  discourse  structure, 
then  consider  summaries  as  parts  of  these  discourse  types,  and  finally  introduce  the  notion  of 
focus.  All  of  these  linguistic  concepts  are  involved  in  the  experimental  design  of  this  study. 

2.1.1  Reasoning 

Reasoning  is  a  discourse  type  in  which  the  speaker  states  a  proposition,  and  then  gives  reasons 
for  believing  its  truth.  The  basic  constructor  of  reasoning  is  STATEMEN  ^/REASON,  which 
connects  two  subtrees  in  a  relation  of  support,  so  that  the  materia;  .-ubordinated  by  the  REASON 
node  provides  logical  support  for  the  material  subordinated  by  the  STATEMENT  node,  as  shown 
in  Figure  1. 


STATEMENT/REASON 


PROPOSITION1  PROPOSITION 

Figure  1;  The  Basic  Constructor  of  Reasoning 

A  STATEMENT/REASON  node  may  embed  other  nodes,  such  as  AND,  OR  and  IF/THEN,  and  either 
it  or  its  subordinates  may  also  embed  further  subordinate  STATEMENT/REASON  nodes.  Thus, 
discourr  'inits  of  this  type  may  have  rather  deep  and  complex  tree  structures. 

Note  that  this  definition  picks  out  a  specific  discourse  type,  which  serves  the  social  function  of 
explanation.  This  social  function  of  explanation  may  be  accomplished  by  many  discourse  types 
including  reasoning,  narrative,  pseudonarrative,  and  planning.  Previous  research  has  studied 
reasoning,  using  as  data  accounts  of  income  tax  decisions,  accounts  of  career  choice,  and 
accounts  of  the  possible  effects  of  certain  political  decisions  (Weiner  79,  Goguen,  Weiner  & 

Linde  83]. 

The  following  is  a  typical  example  of  reasoning  at  the  Boolean  cognitive  level,  from  our 
elicitation  studies;  the  structure  of  this  discourse  is  shown  in  Figure  2. 

You  can  got  a  littla  more  complicated.  Like  you  can  think  about  what 
the  individual  lighte  do.  One  of  the  interesting  things  with  this  one. 


it  doesn’t  depend  upon  this  switch  at  all.  Like  to  see  [inaudible] 
Switch  Two.  So  if  it’s  off  when  Switch  Two  is  down,  it’s  on  when 
Switch  Two  is  up,  independent  of  what  happens  with  Switch  One.  So, 
Switch  Two  controls  light  C.  You  know,  let’s  see,  what's  another  one? 

I  think  D  says  that  the  two  switches  are  the  same  ...  so  if  they’re 
both  down  or  both  up,  D  is  on,  but,  if  they’re  different,  D  is  off.  I 
think  A,  if  I  remember  right,  A  is,  is  th-,  no  B  is  that  they’re  both 
down.  Uh,  in  any  other  position  B  is  off.  And  then  there’s  a  more 
complicated  one.  What  was  that?  If  A’s,  A  says  that  it’s  not  anything 
other  than  One  up  or  Two  down,  then  it’s  on.  Then  that’s  off.  It  gets 
a  little  bit  more  complicated  when  you  explain  it  that  way. 


I/S/E 


Figure  2:  .  Structure  Tree  for  a  Reasoning  Unit 


Notice  in  Figure  2  that  two  nodes  of  this  structure  have  summary  branches;  the  top  (root)  node 
actually  has  two  summary  branches,  one  given  before  the  body  of  the  reasoning  structure  and 
the  other  given  after.  The  body  of  this  reasoning  structure  consists  of  an  AND  that  subordinates 
four  other  reasoning  structures,  one  for  each  light  on  the  box.  A  summary  is  given  after  the 
first  of  these.  It  is  interesting  to  notice  that  the  order  in  which  the  lights  are  considered  is  not 
their  physical  order  on  the  box  (which  would  be  A,  B,  C,  D,  going  left  to  right,  see  Figure  5); 
rather,  the  lights  are  discussed  in  order  of  increasing  logical  complexity  (which  is  C,  D,  B,  A). 

It  is  also  of  interest  to  note  that  some  of  the  REASONS  are  actual  demonstrations  of  what 
happens  when  the  switches  are  in  a  certain  position.  In  one  of  these,  for  the  C  branch,  the 
demonstration  is  given  first,  and  then  three  conclusions  are  drawn  from  it,  subordinated  to  an 
AND  node. 
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2.1.2  Pseudonarrative 

Pseudonarrative  is  a  discourse  type  which  imposes  a  temporal  structure  on  an  essentially  non¬ 
temporal  topic  domain.  It  has  some  but  not  all  of  the  characteristics  of  the  narrative  [Labov 
72,  Linde  &  Labov  75,  Polanyi  85].  Formally,  pseudonarrative  consists  of  an  optional  initial 
summary,  or  "abstract*,  a  sequence  of  past  tense  main  clauses  describing  actions,  and  an 
optional  final  summary,  or  "coda".  Like  narrative,  pseudonarrative  relies  on  the  narrative 
presupposition,  the  rule  of  interpretation  which  states  that  the  order  of  main  clauses  is  to  be 
taken  as  the  order  of  the  events  which  they  describe.  The  difference  is  that  the  past  tense  main 
clauses  of  the  narrative  refer  to  actions  which  are  to  be  understood  as  actually  having 
happened,  while  those  of  the  pseudonarrative  refers  to  hypothetical,  potential,  or  habitual 
actions. 

The  basic  constructor  of  pseudonarrative  is  SEQUENCE,  which  connects  two  or  more  actions  in  a 
relation  of  temporal  sequence  (see  Figure  3). 


SEQUENCE 


ACTION  1 


ACTI0N2 


ACTI0N3 


Figure  3:  Basic  Constructor  of  Pseudonarrative 

The  narrative  presupposition  gives  the  interpretation  of  Figure  3  in  which  the  temporal  order  is 
ACTIONl  precedes  ACTION2  which  precedes  ACTION3.  Since  SEQUENCE  nodes  do  not  self- 
embed,  this  constructor  produces  broad  trees,  with  little  depth.  However,  some  depth  may  be 
provided  by  the  embedding  IF/THEN,  AND  and  OR  nodes  within  ACTION  nodes.  Below  is  a 
pseudonarrative  from  the  elicitation  data: 

We'll  go  through  it  again  slowly.  O.K.  In  one  position  of  these  two 
switches,  where  One  is  up  and  Two  is  down,  all  these  lights  are  off. 

We  take  Switch  One  and  move  it  to  its  other  position,  three  of  the 
lights  come  on.  We  reverse  this  combination  and  make  them  both  go  up, 
we  got  again  three  of  the  lights  are  on  but  a  different  three  lights. 

And,  if  we  move  em  till  both  down,  again,  we  got  three  lights  on  but 
these  two  lights  are  changed  over. 

The  structure  tree  for  this  pseudonarrative  is  given  in  Figure  4.  Note  the  narrative  structure 
imposed  by  the  use  of  the  personal  pronoun  we  and  the  active  verbs  go,  take,  move,  make, 
indicating  actions  by  human  participants  rather  than  states  of  the  device. 
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I/PSEUDONARRATIVE 


We'll  go 
through  it 
again 


SEQ 


Swl 


lights 
come  on 


reverse 

this 


three 

lights 


them  both 
down 


got  three 
lights 


Figure  4:  Structure  Tree  for  a  Pseudonarrative 


Pseudonarrative  has  previously  been  studied  in  the  domain  of  apartment  layout 
descriptions  [Linde  74,  Linde  &  Labov  75].  In  this  domain  the  speakers  commonly  used 
pseudonarrative  structure  to  convert  spatial  organization  to  temporal  organization.  It  was 
found  that  speakers  produced  such  temporally  organized  discourse  units  more  frequently  than 
any  other  way  of  encoding  the  information  of  the  apartment  layout.  Also,  psuedonarratives 
contained  fewer  errors  than  any  other  discourse  type,  suggesting  that  pseudonarrative  is  a 
relatively  simple  discourse  type  for  speakers  to  produce. 

2.1.3  Summaries 

One  important  variable  on  discourse  structure  is  the  presence  or  absence  of  summaries.  A 
summary  is  a  section  of  a  discourse  unit  which  gives  either  the  main  points  of  that  unit,  or  a 
guide  to  the  overall  structure  of  the  discourse  unit,  or  both.  Summaries  have  been  found  to  be 
a  part  of  narrative  structure.  [Labov  72]  calls  initial  summaries  'abstracts"  and  final  summaries 
■codas".  Abstracts,  at  the  beginning  of  narratives,  serve  to  indicate  either  the  narrative’s 
relation  to  the  preceding  discourse,  or  the  way  in  which  the  hearer(s)  should  understand  and 
react  to  the  narrative.  Codas,  found  at  the  end  of  narratives,  serve  either  to  reiterate  the  main 
point,  or  to  bring  the  discourse  back  to  the  present  moment. 


Summaries  are  found  in  both  reasoning  and  pseudonarrative,  and  may  have  either  initial  or 
final  placement.  The  naturalistic  data  elicited  in  the  first  part  of  this  study  contained  a  few 
examples  of  medial  placement,  but  all  were  produced  by  just  one  speaker,  who  was  judged  (by 


independent  criteria)  to  be  outstandingly  disorganized  and  unskillful.  Figures  2  and  4  show 
summaries  in  reasoning  and  psuedonarrative  structures,  respectively. 

Summaries  as  an  integral  part  of  discourse  structure  have  not  previously  been  studied 
experimentally.  However,  their  role  can  be  seen  as  being  in  some  ways  comparable  to  the  role 
of  elaborations  as  discussed  by  [Reder  et  al.  84,  Reder  80],  defined  as 

...  pieces  of  information  that  support,  clarify,  or  further  specify  the  main  points 
(or  gist)  of  a  text. 

2.1.4  Focus 

Another  experimental  variable  depends  on  the  concept  of  linguistic  focus.  Focus  is  a  property 
of  sentences.  The  focus  of  a  sentence  is  that  information  in  the  sentence  which  is  presupposed 
to  be  known  to  the  hearer,  and  to  be  present  to  the  hearer's  attention.  It  is  usually,  but  not 
always,  the  grammatical  subject  of  the  sentence.  In  the  sentences  below,  the  items  in  focus  are 
indicated  in  italics. 

The  lights  depend  on  the  position  of  the  switches . 

The  switches,  you  just  have  to  memorize. 

Focus  was  experimentally  manipulated  in  this  study,  since  rapid  alternation  of  foci  in  adjacent 
sentences  appears  to  produce  passages  that  are  confusing  for  a  reader  or  hearer. 

2.2  Cognitive  Level 

Explanations  of  the  logic  box  can  be  formulated  at  a  number  of  different  cognitive  levels, 
including  the  following: 

1.  Behavioral:  a  simple,  unanalyzed  description  which  matches  a  pattern  of  lights  to  a 
corresponding  switch  position. 

2.  Combinatorial:  treats  the  possible  patterns  of  lights  as  an  aggregate.  This  might  be 
displayed  in  a  table,  like  a  truth  table,  showing  the  relation  of  switch  positions  and  lights. 
(A  possible  addition  at  this  level  of  description  explains  that  only  four  combinations  of 
switch  position  are  possible,  by  simple  multiplication  of  two  switches  times  two  switch 
positions.) 

3.  Logical:  indicates  the  Boolean  functions  of  the  switch  positions  represented  by  each 
light,  using  AND,  OR,  IF,  etc. 

4.  Electronic:  uses  a  circuit  diagram  or  similar  device  to  indicate  how  the  relation  between 
switch  position  and  light  patterns  is  accomplished. 

5.  Physical:  uses  the  principles  of  physics  and  chemistry  that  underly  the  level  of 
electronics. 


The  naturalistic  elicitation  experiments  contain  examples  of  Levels  1  through  3;  we  did  not 
attempt  to  elicit  explanations  at  other  levels. 

Our  experiments  show  (see  Section  4.3)  that  cognitive  level,  as  we  operationalized  it  in  this 
study,  has  a  strong  effect  on  comprehension.  However,  the  notion  of  a  scale  or  hierarchy  of 
conceptual  levels  raises  the  problem  of  scaling.  For  given  tasks,  it  is  often  possible  to  produce 
an  intuitive  or  ad  hoc  scale,  based  solely  on  the  structure  of  the  task  rather  than  on  any 
motivated  theory  of  possible  cognitive  levels.  While  theoretically  motivated  studies  of  cognitive 
level  are  common  in  the  investigation  of  children’s  cognitive  development,  this  research  has  not 
been  extended  to  adult  learning. 

A  number  of  researchers  have  found  that  they  require  cognitive  level  as  a  variable  in  the  study 
of  explanations  in  various  subject  domains.  For  example,  [Kieras  82]  distinguishes  among 
knowledge  of  what  a  device  is  for,  how  to  operate  it,  and  how  it  works.  The  first  two  levels  of 
description  of  our  task  correspond  to  varying  degrees  of  knowledge  of  the  first  type:  how  to 
operate  the  device.  The  third,  fourth,  and  fifth  levels  correspond  to  varying  degrees  of 
knowledge  of  how  the  device  works.  Note  that  a  description  at  any  of  these  five  levels  may 
function  as  an  explanation,  depending  on  the  purpose  of  the  description  and  the  existing  level  of 
understanding  of  the  audience.  Similarly,  in  a  study  of  Navy  instruction  manuals,  [Stevens  & 
Steinberg  81]  provide  a  typology  of  explanations  which  begins  at  the  behavioral  level  and 
proceeds  to  more  abstract  forms  of  explanation.  (It  is  not  possible  to  make  an  exact  match 
between  the  higher  levels  of  their  taxonomy  and  the  taxonomy  given  above,  since  ours  is  a 
simple,  non-branching  tree  structure,  while  theirs  is  a  matrix  of  four  two-way  distinctions. 
However,  there  does  seem  to  be  some  correspondence  in  terms  of  degree  of  abstraction.)  These 
studies,  like  the  current  study,  point  to  the  importance  of  cognitive  level  as  a  variable  in  the 
study  of  comprehension.  A  great  deal  of  further  work  on  adult  learning  is  necessary  to  provide 
a  theoretically  based  understanding  of  possible  cognitive  levels  and  their  effects. 

2.3  Semiotics 

Semiotics  is  a  natural  theoretical  framework  for  the  investigation  of  multimedia  instruction, 
since  it  attempts  to  provide  a  general  theory  of  all  sign  systems  and  their  relations,  including 
spoken  and  written  language,  gesture,  diagram,  etc.  [Goguen  &  Linde  84]  provides  a  general 
formal  theory  for  describing  sign  systems,  providing  a  rich  and  suggestive  vocabulary  for 
formulating  experimental  hypotheses.  Appendix  I  summarizes  aspects  of  this  general  theory. 
The  following  discusses  some  semiotic  concepts  which  were  used  in  formulating  the  hypotheses 
of  this  study. 


MBBWBB  )ftfaaam  SXKMMUTI  *xxmA»  mokkkxs 


36%3ghi 


_ ^rz 


H 


11 

2.3.1  Cross-media  Reference  and  Visnal  Deixis 

Multimedia  instruction  typically  involves  mixed  sign  systems,  that  is,  its  signs  occur  in  a 
mixture  of  media  (eg.  visual  and  verbal,  or  verbal  and  gestural).  Under  such  circumstances,  it 
is  important  to  coordinate  events  happening  in  different  media.  A  phenomenon  of  particular 
importance  for  this  kind  of  coordination  is  the  use  of  a  sign  in  one  medium  to  indicate  a  sign  in 
another;  we  call  this  cross-media  reference.  It  is  more  pervasive  than  might  at  first  be 
supposed.  In  our  experiments,  we  found  that  instructors  often  point  at  things  they  have  written 
or  drawn  on  the  board.  The  material  on  the  board,  in  turn,  may  refer  to  properties  or  states  of 
the  physical  logic  box.  Moreover,  an  instructor’s  spoken  language  often  refers  to  aspects  of  the 
box  or  to  material  on  the  board. 

One  example  of  cross-media  reference  in  the  experimental  domain  is  a  reference  to  some  aspect 
of  a  diagram,  which  itself  refers  to  some  aspect  of  the  states  of  the  box,  as  in 

The  second  line  shows  what  happens  when  both  of  the  switches  are  down. 

For  a  more  complex  example,  an  instructor  in  one  of  our  elicitation  sessions,  on  discovering  that 
the  box  was  now  behaving  in  a  way  that  seemed  inconsistent  with  its  previous  behavior,  pointed 
at  a  table  on  the  blackboard  and  said 
So  it  should  have  been.  C  on. 

Here  both  it  and  the  gesture  refer  to  a  row  of  the  table,  which  in  turn  refers  to  a  state  of  the 
box.  (This  reference  had  been  previously  established  through  a  combination  of  spoken  language 
and  actually  operating  the  box.)  It  might  have  been  difficult  or  impossible  to  resolve  the 
referent  of  it  without  the  gesture. 

Cross-media  reference  is  often  used  in  natural  language  for  emphasis,  that  is,  to  point  out  or 
emphasize  certain  words.  In  written  language,  choice  of  type  font  is  generally  considered  a 
completely  independent  medium  from  that  of  language.  Therefore,  choice  of  font  can  be,  and 
often  is,  used  for  emphasis.  Typical  font  choices  for  this  purpose  are  ALL  CAPS,  bold,  and 
italics.  The  relevance  of  a  systematic  study  of  cross-media  reference  to  computer-based 
instruction  is  evident:  one  would  like  to  know  the  optimal  ways  to  use  emphasis  in  an 
instructional  context.  For  example,  one  might  put  parts  of  a  text  (in  a  window  on  a  screen)  in 
reverse  contrast;  or  one  might  let  it  flash;  or  put  it  in  a  different  color  or  a  different  size,  or  a 
different  font;  or  some  combination  of  the  above.  Similar  consideration  apply  to  graphics, 
animations,  icons,  sound  effects,  and  computer-generated  speech.  One  must  decide  what  to 
emphasize,  and  how  to  emphasize  it,  in  order  to  hold  together  an  ongoing  stream  of  information 
in  a  mixed  media  system.  Too  much  emphasis,  or  too  many  different  kinds  of  emphasis,  will 
only  produce  confusion.  These  theses  should  also  apply  to  more  traditional  media,  such  as 
overhead  projections  and  videotape. 


The  most  important  kind  of  cross-media  reference  in  this  investigation  is  visual  deixls,  defined 
as  the  use  of  gesture  to  refer  to  some  aspect  of  the  visual  world  in  order  to  connect  it  with  some 
element  of  concurrent  spoken  or  written  language.  Note  that  this  restricted  definition  excludes 
many  instances  of  pointing  or  visual  reference  when  there  is  no  language  actually  present.  The 
definition  also  excludes  situations  in  which  a  speaker  points  to  some  aspect  of  the  visual  scene 
which  is  unconnected  to  his  ongoing  talk  (for  example,  gesturing  to  point  out  the  passage  of  an 
acquaintance  during  the  course  of  an  unrelated  conversation).  Visual  deixis  as  defined  here  has 
not  previously  been  studied;  previous  studies  of  gesture  have  concentrated  on  independent 
gestures  which  are  not  keyed  to  features  of  the  concurrently  present  physical  world  (Efron 
41,  McNeill  79]. 

2.3.2  Levels  of  Discourse  Structure  and  Sign  Systems 

The  semiotic  theory  summarized  in  Appendix  I  suggests  the  importance  for  comprehension  of 
hierarchical  organization  in  complex  sign  systems.  In  natural  language,  every  level  (except  the 
lowest  level  of  phonetics)  is  composed  of  sequences  of  units  from  the  next  lower  level;  the 
hierarchy  is  in  fact  a  multilevel  whole/part  hierarchy.  These  general  considerations  (e.g.,  see 
the  discussion  of  "semiotic  morphism"  in  Appendix  I)  suggest  that  it  should  be  more  important 
to  preserve  the  higher  level  structures  than  the  lower  level  structures.  Reflecting  their  relative 
importance,  linguistic  entities  that  mark  the  boundaries  of  higher  level  structures  tend  to  be  less 
subtle,  more  obvious. 

2.4  Experimental  Variables  Suggested  by  These  Theories 

The  theories  discussed  above  provide  a  foundation  for  identifying  a  number  of  key  variables 
used  in  hypotheses  about  effective  multimedia  instruction.  These  variables  include: 

1.  Discourse  type.  This  refers  to  the  linguistic  structure  of  the  explanation;  the  two 
discourse  types  we  studied  are  Pseudonarrative  and  Reasoning. 

2.  Structure  Markers.  Linguistic  markers  of  internal  discourse  structure  may  or  may  not 
be  employed. 

3.  Visual  Deixis.  Gestures  can  connect  visual  and  verbal  media.  We  manipulated 
occurrence  of  pointing  to  some  aspect  of  the  logic  box,  or  the  diagram  used  to  explain  it. 

4.  Cognitive  level.  The  specific  levels  we  explored  are  Combinatorial  and  Boolean. 

5.  Summaries.  Initial  and/or  final  summaries  may  be  present  or  absent  in  a  given 
explanation.  (We  did  not  examine  effects  of  medial  summaries.) 

6.  Within-Sentence  Focus  Consistency.  The  focus  (on  switches  or  lights)  may  be 
consistent  or  inconsistent  within  parallel  structures  within  sentences. 

7.  Discourse  Unit  Boundary  Violations.  Related  information  may  or  may  not  be 
continued  across  the  boundaries  of  discourse  subunits. 
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8.  Discourse  Unit  Scrambling.  The  ordering  of  discourse  units  may  be  permuted  or  not. 

9.  Consistency  of  Visual  Icon.  The  diagram  may  be  coordinated  or  uncoordinated  with 
the  verbal  explanation  that  accompanies  it. 

3  Method 

This  section  discusses  the  experimental  task,  experiment  design,  choice  of  subjects,  production 
of  test  stimuli,  and  dependent  measures. 

3.1  The  Experimental  Task 

The  substantive  task  utilized  in  this  research  involves  a  “logic  box"  (see  Figure  5).  This  box 
has  two  switches  and  four  lights,  each  light  being  a  logical  function  of  the  positions  of  the  two 
switches.  Both  in  the  study  of  elicited  explanations,  and  in  the  experimental  study,  the  box  was 
taken  to  be  the  controller  for  an  irrigation  system,  which  is  able  to  deliver  different  mixtures  of 
water  and  chemicals  depending  upon  the  setting  of  its  switches. 
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3.2  Hypotheses  and  Experimental  Design 

The  hypotheses  were  grouped  into  two  series:  one  a  fully-crossed  design  on  three  binary 
independent  variables,  and  the  other  a  set  of  six  individual  variables.  The  first  series,  the 
Combinatorial  Series,  consists  of  explanations  presented  at  the  Combinatorial  level, 
accompanied  by  a  referentially  coordinated  visual  representation.  The  second  series,  the 
Boolean  Series,  consists  of  explanations  presented  at  the  Boolean  level,  all  but  one  of  which  is 
accompanied  by  a  referentially  coordinated  visual  representation.  In  the  Boolean  level 
explanations,  the  box  was  explained  to  be  programmable;  that  is,  its  current  relation  between 
lights  and  switches  was  taken  to  be  only  one  of  many  possible.  In  the  Combinatorial  level 
explanations,  the  box  was  explained  as  non-programmable;  that  is  the  relation  between  lights 
and  switches  was  taken  to  be  fixed. 

3.2.1  Explanations  at  the  Combinatorial  Level:  Research  Design 

Described  below  are  eight  conditions  representing  explanations  at  the  Combinatorial  level. 
These  eight  conditions  comprise  a  fully-crossed  design  with  three  binary  independent  variables 
which  varies  Discourse  Type  (DT,  Pseudonarrative  versus  Reasoning),  Visual  Deixis  (VD, 
present  or  absent)  and  verbal  Discourse  Structure  Marking  (SM,  present  or  absent),  as  shown  in 
Figure  9.  The  texts  for  these  stimulus  conditions  are  given  in  Appendix  II.  All  these 
explanations  assume  that  the  configuration  of  the  box  is  fixed,  rather  than  programmable,  and 
all  employed  the  diagram  shown  in  Figure  12. 
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Figure  6:  The  Logic  Box 

3.2.2  Explanations  at  the  Boolean  Level:  Research  Design 

The  second  series,  at  the  Boolean  cognitive  level,  has  as  its  main  discourse  unit  an  explanation 
which  has  Reasoning  structure  rather  than  Pseudonarrative  structure.  The  overall  strcuture  of 
this  stimulus  consists  of  a  plan  structure  whose  goal  is  stated  in  the  initial  task  description: 

Today  we’re  going  to  learn  how  to  operate  this  device 
.  The  plan  consists  of  a  number  of  discourse  units  embedded  under  the  plan  node,  including 
explanation  structures,  a  description,  and  initial  and  final  summaries.  The  top-level  discourse 
structure  of  this  stimulus  is  shown  in  Figure  6. 

A  "standard*  explanation  with  this  structure,  and  six  variants  of  it  were  employed: 

Bl:  Standard  Boolean  explanation,  including  initial  and  final  summaries,  a  coordinated  visual 
icon,  and  internally  consistent  subunits; 

B2:  No  Initial  summary;2 


“When  summaries  were  omitted,  appropriate  filler  material  was  included  to  hold  length  approxiamtely  constant. 
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Figure  8:  Top-level  Structure  of  Boolean  Level  Stimulus 


B3:  No  Final  summary; 

B4:  Sentential  focus  inconsistencies; 

B5:  Discourse  subunit  boundary  violations; 

B6:  Scrambled  ordering  of  discourse  subunits; 

B7:  Use  of  diagram  not  coordinated  with  the  cognitive  level  of  the  verbal  portion  of  the 
stimulus.  (A  combinatorial  rather  than  a  Boolean  level  diagram  was  used.) 


We  may  note  that  B-5  was  constructed  from  B-l  by  permuting  clauses  across  discourse  unit 
boundaries,  and  then  modifying  intersentiential  connectives  to  make  the  language  smoother. 
B-6  was  constructed  from  B-l  by  permuting  entire  component  discourse  units. 


The  texts  for  these  stimulus  conditions  are  given  in  Appendix  H.  All  these  explanations  assume 
that  the  configuration  of  the  box  is  programmable. 


3.3  Subjects 

The  experimental  subjects  were  students  at  Foothill  College,  a  local  community  college. 
Subjects  were  volunteers,  chosen  to  have  no  college  background  in  mathematics,  logic,  or 
computer  science,  and  to  be  native  speakers  of  English.  They  received  $10  for  participating. 
Bout  25  subjects  per  condition  were  tested. 


3.4  Stimulus  Production 

The  explanations  described  above  were  videotaped.  A  professional  actor  was  used  to  ensure 
word-perfect  delivery,  as  well  as  consistency  of  delivery,  gesture,  and  timing  across  conditions. 
The  actor  partially  memorized  and  partially  read  from  scripts  (projected  onto  the  wall  behind 
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the  camera),  since  the  perfect  delivery  of  scripts  which  differ  only  slightly  is  very  difficult.  In 
order  to  reduce  variations  of  the  Visual  Deixis  variable,  the  actor  used  little  or  no  gesture  or 
movement  other  than  those  specified  by  the  script.  This  led  to  a  somewhat  stiff  delivery,  which 
was  noted  by  some  of  the  subjects. 

The  explanations  were  filmed  using  a  single  stationary  video  camera.  A  stationary  camera, 
with  no  change  of  camera  angle  or  focal  length,  was  used  to  avoid  introducing  the  additional 
form  of  deixis  which  would  be  imposed  by  camera  work.  Wherever  possible,  segments  repeated 
across  conditions  were  filmed  separately  and  edited  in.  In  order  to  facilitate  such  editing,  the 
scripts  were  divided  into  short  scenes,  and  the  actor  walked  off  and  then  back  on  camera 
between  these  scenes. 

3.5  Dependent  Measures 

Following  the  video-taped  explanation  of  how  logic  boxes  work,  a  paper-and-pencil  test  was 
administered  by  the  experimenter.  This  was  an  objective  test,  with  12  items  for  the 
Combinatorial  Series  and  16  items  for  the  Boolean  Series.  The  tests  have  7  items  in  common  to 
enable  cross-series  comparisons.  Students  were  permitted  to  take  notes  during  the  explanation, 
and  to  use  these  notes  during  the  test,  since  our  pilot  tests  showed  that  forbidding  the  use  of 
notes  produced  extremely  low  scores  with  very  little  variance.  These  tests  are  reproduced  in 
Appendix  III. 

4  Analytic  Procedures  and  Results 

This  section  summarizes  the  statistical  results  from  this  research,  along  with  the  analytical 
procedures  that  produced  them  and  some  discussion  of  their  meaning.  Section  5  gives  a  more 
complete  interpretation  of  the  results.  Recall  that  the  general  structure  of  the  experiments 
involves  two  series  of  stimuli,  hereafter  called  the  C  or  Combinatorial  Series  and  the  B  or 
Boolean  Series.  Subsections  below  discuss  general  issues,  present  the  procedures  and  results  for 
the  C  Series  data,  and  then  for  the  B  Series  and  cross-series  data. 

4.1  General  Considerations 

This  subsection  discusses  level  of  significance,  gives  an  overveiw  of  the  C  and  B  datasets,  and 
then  considers  questionnaire  item  choice. 


4.1.1  Levels  of  Significance 

This  project  is  an  initial  study  in  the  largely  unexplored  field  of  multi-media  explanation 
structure,  using  a  new  research  method  with  new  questionnaires  as  dependent  measures,  and 
based  upon  relatively  novel  insights  from  linguistics  and  semiotics.  Hence,  it  should  be  regarded 
as  paradigmatic  and  exploratory.  We  regard  a  result  as  "significant"  if  its  obtained  probability 
level  is  less  than  or  equal  to  .05,  and  as  "marginally  significant  if  between  .05  and  .10;  results 
with  a  probability  level  less  than  or  equal  to  .01  will  be  called  "very  significant."  In  most  cases, 
the  obtained  probability  value  is  also  given,  so  that  the  reader  can  independently  judge  the 
significance  of  the  results.  Two  types  of  statistical  test  are  employed:  F  tests  for  analyses  of 
variance,  and  t  tests  for  between-group  comparisons.  Unless  otherwise  indicated  (e.g.,  in  testing 
directional  hypotheses)  two-tailed  tests  are  used. 

4.1.2  Choice  of  Questionnaire  Items 

The  C  Series  questionnaire  has  12  items'  ,  the  B  Series  has  16  items,  and  these  two  instruments 
have  7  items  in  common.  Since  none  of  these  items  had  previously  been  used  in  experimental 
studies,  it  would  have  been  desirable  to  do  prior  item-  and  scale-reliability  studies; 
unfortunately  this  was  impractical  within  the  limits  of  the  present  project.  However,  we 
computed  the  correlation  of  each  item  with  questionnaire  score  totals  for  each  stimulus 
condition  (in  its  series),  and  also  across  all  stimuli  (in  its  series),  in  order  to  determine  the  extent 
to  which  each  question  was  contributing  useful  information  to  the  analysis.  The  worst  item 
that  we  found,  item  14  in  the  B  Series,  showed  low  or  negative  correlation  with  total  scores  in  3 
stimulus  conditions;  it  had  a  .11  average  correlation  with  total  scores  across  the  B  Series, 
attaining  a  reasonably  high  r  value  in  some  conditions.  Since  analytic  results  item  14  differed 
very  little  from  those  that  included  item  14,  its  exclusion  did  not  seem  warranted. 

Consequently,  we  used  all  items  in  computing  the  scores.  Tables  of  correlations  are  given  in 
Figures  7  and  8. 

4.2  Combinatorial  Series  Results 

The  C  Series  experiments  studied  effects  of  three  2-level  independent  variables,  Discourse  Type, 
Visual  Deixis,  and  Structure  Marking.  The  means  and  standard  deviations  for  test  scores 
obtained  in  the  resulting  conditions  are  given  in  Figure  9. 

Combinatorial  explanation  data  were  analyzed  using  the  SAS  statistical  package  3-way  analysis 
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Item  5  of  the  C  Series  questionnaire  was  excluded  because  it  was  administered  in  an  inconsistent  manner; 
otherwise,  there  would  have  been  13  items,  as  given  in  Appendix  III.l. 
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Figure  7:  Correlations  between  C  Series  Questionnaire  Items  and  Total  Scores,  by  Condition 


uestionl 


17 


mean  r 


B2  I 


.18 


.37  I-. 


.35  I  . 


Condition 
14  1 


I  47 


.67  | 


I  .59  | 


1  .30 

1  -6 

9 

.19  1  .33  !  .15 

1  • 

48  |  .27 

.19 


.40  I 


.21  I 


.38  | 


.12  I 


.07  I- 


.36 _ I 


.26  I 


.01  I- 


.32  I 


.38  I 


.40 


.25  I 


23  I 


I  .34  .34 


I  .22  I  .42 


I  .29  |  .09 


I  .41  |  .68 


I  .23  I  .36 


I  .16  I  .46 


I  .17  I  .75 


I  - . 22  |  .42 


I  .29  |  .08 


I -.009  I  .19 


I -.003  |  .29 


1  .25  |  .32 


I  22  I  26 


|  mean  I 


.50  I  .33  | 


04  I  .26  | 


.02  I  .27  | 


38  |  .35  |  .40  | 


07  |  .36  |  .24  I 


48  .32  I  .19  | 


53  |  .53  |  48  | 


38  |  .002  I  .18  | 


25  |  .02  I  .11  I 


07  |  .30  I .25 


25  |  .50  1  .27  | 


30  .24  I  .27 


25  I  21  I 


Figure  8:  Correlations  between  B  Series  Questionnaire  Items  and  Total  Scores,  by  Condition 

of  variance  program  (ANOVA).  Total  score  on  the  12-item  C  Series  questionnaire  as  dependent 
measure.  Our  hypothesis  was  that  each  variable  would  significantly  enhance  comprehension 
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Figure  8:  Combinatorial  Series  Descriptive  Statistics 


scores.  The  overall  model  tested  achieves  a  value  of  F=3.19  (df=4,204  with  p=.0144).  Main 
effects  from  two  independent  variables  were  observed.  Discourse  Type  yields  a  highly 
significant  effect  on  comprehension  (F=7.00,  with  df=l,208  and  p=.0088),  subjects  in  the 
pseudonarrative  condition  scoring  systematically  higher  than  those  in  the  reasoning  condition. 
Presence  of  Visual  Deixis  has  a  marginally  positive  influence  on  dependent  variable  values 
(F=3.67,  with  df=l,208  and  p=.057).  The  third  independent  variable,  Sturcture  Marking, 
does  not  significantly  affect  comprehension  test  outcomes.  Finally,  none  of  the  interaction 
terms  in  the  model  is  significant. 

What  this  means  is  that  the  largest  single  effect  on  comprehension  of  explanations  is  from 
Discourse  Type,  with  Pseudonarrative  being  very  significantly  better  than  Reasoning  for  our 
experimental  task.  Visual  Deixis  also  has  a  positive  effect  on  the  dependent  measure.  The 
third  most  significant  is  the  interaction  between  Visual  Deixis  and  Structure  Marking. 

Structure  Marking  is  not  significant,  but  contributes  to  the  VD*SM  interaction,  which  is  more 
significant  than  the  SM  main  effect.  This  indicates  that  Structure  Marking  has  complex  effects, 
and  is  not  simply  additive. 

Pairwise  comparison  between  subject  groups’  comprehension  scores  using  Student’s  t  test  helps 
to  clarify  the  situation.  One  immediate  result  is  that  the  most  effective  condition  is 
Pseudonarrative  with  Visual  Deixis  and  Structure  Marking,  the  condition  in  which  all  factors 
hypothesized  to  facilitate  comprehension  are  present  (l-tailed  t  tests  show  this  to  be  significant 
for  all  conditions  except  Pseudonarrative  with  Visual  Deixis  but  without  Structure  Marking). 
Similarly,  the  least  effective  condition  is  Reasoning  without  Visual  Deixis  and  without  Structure 
Marking,  in  which  all  hypothetically  facilitating  factors  are  absent.  This  seems  reasonable. 


The  hypothesis  that  the  mean  of  Pseudonarrative  with  Visual  Deixis  and  Structure  Marking  is 
greater  than  the  mean  of  Pseudonarrative  with  Visual  Deixis  but  without  Structure  Marking, 


has  t=1.96,  which  is  (1-tailed)  significant;  i.e.,  in  Pseudonarrative  with  Visual  Deixis,  Structure 
Marking  facilitates  comprehension.  On  the  other  hand,  we  obtain  a  t  value  1.83  for  the 
hypothesis  that  the  mean  of  Pseudonarrative  without  Visual  Deixis  and  without  Structure 
Marking  is  greater  than  the  mean  of  Pseudonarrative  without  Visual  Deixis  but  with  Structure 
Marking.  This  is  (1-tailed)  significant;  i.e.,  in  Pseudonarrative  without  Visual  Deixis,  adding 
Structure  Marking  inhibits  comprehension.  This  helps  explain  why  Structure  Marking  does  not 
show  a  main  effect,  but  instead  appears  in  an  interaction  term. 

The  results  for  Visual  Deixis  lean  in  the  same  directions,  but  the  interactions  are  less  significant. 
The  hypothesis  that  the  mean  of  Pseudonarrative  without  Visual  Deixis  and  without  Structure 
Marking  is  greater  than  the  mean  of  Pseudonarrative  with  Visual  Deixis  but  without  Structure 
Marking  says  that  in  Pseudonarrative  without  Structure  Marking,  adding  Visual  Deixis  might 
make  things  worse  (its  t=.81  is  definitely  not  significant).  In  contrast,  the  hypothesis  that  the 
mean  of  Pseudonarrative  with  Visual  Deixis  and  with  Structure  Marking  is  greater  than  the 
mean  of  Pseudonarrative  without  Visual  Deixis  but  with  Structure  Marking  says  that  in 
Pseudonarrative  with  Structure  Marking,  adding  Visual  Deixis  definitely  helps  (its  t=2.99  is 
very  significant).  Thus,  it  is  reasonable  that  Visual  Deixis  should  have  a  main  effect. 


4.3  Boolean  Series  Results 

The  B  Series  consists  of  7  individual  stimuli,  described  with  their  means  and  standard 
deviations  in  Figure  10. 
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Figure  10:  Boolean  Series  Descriptive  Statistics 


The  seven  Boolean  Series  stimuli,  all  of  which  employ  Reasoning  discourse  structure  at  the 
Boolean  cognitive  level,  were  arranged  into  a  variety  of  patterns  and  then  examined  using  total 
score  on  the  16  item  B  Series  questionnaire  as  dependent  measure  with  t  tests  and  1-way 
ANOVAs.  The  means  and  standard  deviations  for  these  7  stimuli  are  given  in  Figure  10.  The 
two  most  evident  and  significant  results  are  as  follows:  First,  the  condition  which  has  initial 


summary  replaced  by  filler  is  the  least  effective  condition;  this  means  that  among  the 
manipulations  studied  here,  the  strongest  effect  is  the  omission  of  the  initial  summary.  Second, 
the  condition  using  a  diagram  at  the  Combinatorial  rather  than  the  Boolean  level  is  the  best 
stimulus;  this  is  (2-tailed)  significant  for  all  stimuli  except  Sentential  Focus  Inconsistencies  and 
Discourse  subunit  boundary  violations.  As  discussed  in  Section  5,  this  result  demonstrates  the 
importance  of  relevantly  iconic  graphics  for  effective  instruction. 

We  tested  for  the  effect  of  cognitive  level  with  a  1-way  3-level  ANOVA,  using  data  from  both  the 
Combinatorial  and  Boolean  Series,  using  as  the  dependent  measure  the  common  items  in  the 
questionnaires  for  the  two  series.  The  specific  hypothesis  is  that  combinatorial  level  reasoning 
structure  is  more  effective  than  boolean  level  reasoning  structure  using  a  combinatorial 
diagram,  which  is  more  effective  than  boolean  level  reasoning  structure  using  a  Boolean 
diagram.  This  result  very  significant  (F=13.18,  df=2,68  with  p=.0001).  We  conclude  that  the 
combinatorial  level  is  much  more  effective  for  this  task,  and  that  even  introducing  it  in  a 
diagram  without  coordinating  it  with  the  discourse  is  helpful. 

In  order  to  test  whether  manipulations  of  summaries  or  of  structural  boundary  violations  had 
more  effect,  a  I-way  2-level  ANOVA  was  used  to  compare  two  groups  of  stimuli:  the  stimuli 
involving  deformations  of  discourse  structure  at  various  levels  of  structure,  and  those  involving 
replacement  of  initial  and  final  summaries.  The  hypothesis  is  that  replacement  of  a  summary 
has  a  greater  effect  on  comprehension.  This  comparison  yielded  F=8.46  (df=  1,162  with 
p=.0044),  which  is  very  significant. 

A  further  hypothesis  on  the  role  of  summaries  is  that  having  both  initial  and  final  summary  is 
best,  and  having  a  relevant  initial  summary  is  more  helpful  than  having  a  relavant  final 
summary.  To  test  this,  a  1-way  3-level  ANOVA  was  used  to  compare  the  mean  of  the  standard 
Boolean  level  stimulus  having  inital  and  final  summaries  with  the  mean  of  the  stimuli  with 
Initial  Summary  replaced  by  filler  and  Final  Summary  replaced  by  filler.  This  result  is  not 
significant.  However,  analysis  of  variance  is  not  the  best  way  to  reveal  ordering  among 
conditions.  On  the  other  hand,  our  initial  theoretical  framework  would  not  have  justified  a 
directional  hypothesis  in  advnace.  (Primacy  effects  literature  would  predict  suppression  of  the 
initial  summary  would  have  more  impairing  effects  while  recency  literature  might  predict  lower 
comprehension  when  the  final  summary  is  suppressed.)  Consequently  we  made  pairwise 
comparisons  between  the  conditions  using  t  statistics.  In  this  case,  the  Boolean  level  stimulus 
with  initial  and  final  summary  is  slightly  more  effective  than  than  the  stimulus  having  initial 
summary  replaced  by  a  filler.  This  result  is  marginally  significant  using  a  1-tailed  t  test 
(t=1.55). 


Looking  at  the  set  of  stimuli  involving  discourse  unit  boundary  violations,  the  means  shows  the 
following  ordering:  the  stimulus  with  Focus  Inconsistency  is  more  effective  than  the  one  with 
Discourse  Unit  Boundary  Violations,  which  is  more  effective  than  the  one  with  Scrambled 
Scenes.  Thus,  larger  the  scope  of  the  discourse  structure  boundaries  violated,  the  greater  is  the 
effect.  Once  again,  a  1-way  3-level  ANOVA  does  not  show  this  pattern  as  significant.  However 
a  comparison  of  Focus  Inconsistency  with  Scrambled  Scenes  is  significant  (1-tailed  t=1.83), 
which  adds  to  the  suggestive  evidence  for  this  finding.  Moreover,  this  "Breadth  of  Scope 
Hypothesis*  is  consistent  with  the  earlier  result  that  summaries  have  more  drastic  effects. 
Summaries  may  be  thought  of  as  structure  markings  as  representing  the  largest  possible  scope, 
since  they  mark  the  beginning  and  ending  of  reasoning  structures,  like  opening  and  closing 
parentheses  for  an  expression,  (note:  earlier  version  of  this  section  is  file  called  results.mss 

5  Discussion  and  Conclusions 

This  section  interprets  the  results  presented  in  the  previous  section;  this  yields  some  suggestions 
for  future  research  and  some  recommendations  for  the  practical  production  of  multimedia 
instructional  material. 

5.1  Combinatorial  Level 

We  begin  with  a  discussion  of  the  results  of  the  statistical  tests  on  data  from  the  Combinatorial 
Series,  in  which  the  patterns  of  lights  are  explained  as  the  result  of  the  patterns  of  switch 
positions. 

5.1.1  Effect  of  Discourse  Type 

The  strongest  finding  is  that  discourse  type  has  an  important  effect  on  comprehension.  Of  the 
possible  discourse  types,  we  tested  pseudonarrative  and  reasoning.  These  were  the  two 
discourse  types  most  frequently  found  in  our  elicitations  of  naturalistic  explanations,  and 
therefore,  are  frequently  represented  in  classroom  instruction.  Pseudonarrative  was  found  to  be 
much  more  effective  for  the  subject  matter  and  trial  instrument  used  in  this  study. 

This  is  an  entirely  new  finding.  While  previous  research  on  instruction  has  investigated  many 
variables  both  at  a  micro-level  (e.g.,  frequency  index,  effect  of  sensical  versus  nonsensical 
stimuli,  forward  and  backward  transfer,  primacy  and  recency  effects,  etc.)  and  at  a  macro-level 
(e.g.,  behavioral  modelling,  prior  set,  conformity  to  expectation,  etc.)  the  effect  of  discourse 
type  has  not  been  considered.  This  finding  is  important  because  it  shows  that  even  if  cognitive 
level,  length,  and  information  content,  which  are  known  to  affect  comprehension,  are  held 
constant,  a  strong  difference  can  still  be  produced  by  choice  of  discourse  organization.  The 


immediate  practical  value  of  this  finding  is  to  suggest  pseudonarrative  as  a  preferred 
organization  for  instructional  discourse,  particularly  with  novices.  This  conclusion  should  be 
tested,  however,  for  generalizability  with  other  explanations.  The  finding  also  suggests  the 
potential  value  of  much  more  extensive  research  on  the  effects  on  comprehension  of  a  wide 
range  of  discourse  types,  and  of  variations  of  structures  within  discourse  types. 

There  are  a  number  of  observations  to  be  made  about  the  effect  of  pseudonarrative.  In  order  to 
understand  why  pseudonarrative  facilitates  comprehension  more  than  reasoning,  we  may  look 
first  at  the  basic  constructors  of  these  discourse  types.  As  mentioned  in  Section  2.1.2,  the  basic 
construtor  of  pseudonarrative  is  SEQUENCE,  which  connects  two  or  more  actions  in  a  relation  of 
temporal  order.  Since  SEQUENCE  nodes  do  not  self-embed,  this  constructor  tends  to  produce 
broad  trees.  In  contrast,  the  basic  constructor  of  reasoning  is  STATEMENT/REASON,  connecting 
a  statement  about  the  state  of  the  world  with  a  reason  for  believing  it.  STATEMENT/REASON 
nodes  may  embed  other  constructor  nodes  in  complex  ways,  including  other 
STATEMENT/REASON  nodes,  so  that  trees  of  considerable  depth  are  possible.  As  [Yngve  60]  has 
shown,  excessive  tree  depth  can  produce  considerable  difficulties  in  language  processing 
(although  note  that  his  findings  are  at  the  sentence  level  rather  than  the  discourse  level). 

It  might  be  argued  that  the  fundamental  difference  between  pseudonarrative  and  reasoning  lies 
not  in  discourse  structure,  but  rather  in  the  fact  that  pseudonarrative  involves  the  actions  of 
persons,  and  is  hence  more  interesting  and  involving  than  reasoning,  which  focuses  on  objects 
and  events.  This  can  be  investigated  by  studying  the  distribution  of  sentential  subjects  in  the 
experimental  stimuli  used  in  these  experiments.  In  fact,  in  the  stimuli  we  employed,  both 
pseudonarrative  and  reasoning  employ  both  types  of  sentential  subjects  (see  Figure  11). 

Although  pseudonarrative  has  more  personal  subjects,  the  distribution  of  differences  is  not 
significant  (using  the  £2  statistic).  This  suggests  that  discourse  structure  rather  than  type  of 
sentential  subject  is  responsible  for  the  observed  differences  in  comprehension.  It  would  be 
interesting  to  explore  this  issue  more  fully  by  keeping  subject  types  constant  and  varying 
discourse  structure,  and  vice  versa. 


Personal 

Impersonal 

Sublecte 

Subl ecte 

Pseudonarrative 

18 

13 

Reasoning 

6 

11 

Figure  11:  Sentential  Subject  Types  in  Pseudonarrative  and  Reasoning 

Explanations 


Other  research  has  also  found  pseudonarrative  a  preferred  discourse  type.  [Linde  74]  found  that 
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96%  of  speakers  gave  a  pseudonarrative  descriptions  of  apartment  layouts;  the  only  alternative 
found  was  map-structured  descriptions.  Further,  the  pseudonarrative  descriptions  contained  far 
fewer  speech  errors  and  false  starts  than  the  map  descriptions.  This  is  confirming  evidence, 
from  the  domain  of  production  rather  than  perception,  that  pseudonarrative  is  better  for 
certain  tasks. 

Pseudonarrative  structure  is  essentially  the  same  as  the  structure  of  oral  narrative.  Oral 
narrative  is  known  to  be  one  of  the  most  basic  forms  of  discourse  organization.  It  is  among  the 
first  discourse  types  that  children  develop.  (Only  response  pairs  such  as  question-answer  and 
greeting-greeting  precede  it.)  Moreover,  narrative  serves  as  the  model  for  a  wide  variety  of 
discourse  types  such  as  pseudonarrative,  hypothetical  narrative,  task  directions,  route 
directions,  etc.  Thus  it  should  not  be  surprising  that  pseudonarrative  is  often  more  effective 
than  other  althernatives  for  discourse  organization. 

5.1.2  Effect  of  Visual  Deixis 

Recall  that  Visual  Deixis  is  defined  as  the  use  of  gesture  to  refer  to  some  aspect  of  the  visual 
world.  In  these  experiments,  such  reference  is  generally  either  to  the  logic  box  itself,  or  to  the 
diagram  on  the  board.  A  3-way  ANOVA  on  C  Series  scores  found  that  Visual  Deixis  has  a 
marginally  significant  effect  on  comprehension,  and  is  the  second  most  important  component  of 
a  significant  model.  Moreover,  t  tests  showed  that  Visual  Deixis  has  a  very  significant 
facilitating  effect  in  Pseudonarrative  when  Structure  Marking  is  already  present.  However,  t 
tests  showed  no  significant  effects  for  Visual  Deixis  in  Reasoning.  This  may  be  because  gesture 
traditionally  forms  an  integral  and  expected  part  of  the  Narrative  discourse  type,  which  is  the 
prototype  for  Pseudonarrative,  whereas  gesture  is  less  important  for  the  more  abstract 
Reasoning  discourse  type.  The  question  of  the  role  of  visual  deixis  relative  to  different 
discourse  types  clearly  requires  more  research. 

5.1.3  Effect  of  Structure  Marking 

Even  though  the  3-way  ANOVA  did  not  find  a  significant  main  effect  for  Structure  Marking,  t 
tests  show  that  it  plays  a  significant  but  complex  role.  In  Pseudonarrative,  Structure  Marking 
has  a  significant  facilitating  effect  when  Visual  Deixis  was  already  present,  but  has  a  significant 
negative  effect  in  the  absence  of  Visual  Deixis.  In  Reasoning,  Structure  Marking  showed  no 
significant  effects.  The  difference  in  these  cases  may  be  explained  by  the  nature  of  discourse 
structure  in  the  two  discourse  types.  Pseudonarrative  structure  (see  Section  2.1.2)  is  simpler 
than  Reasoning  structure,  so  that  cues  to  the  structure  may  be  more  confusing  if  they  occur  in 
unnatural  forms,  such  as  Structure  Marking  without  Visual  Deixis. 


It  is  a  somewhat  puzzling  result  that  Pseudonarrative  without  Visual  Deixis  and  with  Structure 
Marking  is  not  more  effective  than  Reasoning  without  Visual  Deixis  and  with  Structure 
Marking,  or  than  Reasoning  without  Visual  Deixis  and  without  Structure  Marking.  One  would 
expect  among  stimuli  not  having  Visual  Deixis  but  having  Structure  Marking,  that  changing  for 
Reasoning  to  Pseudonarrative  would  yield  a  significant  improvement  in  comprehension,  since 
Discourse  Type  has  the  strongest  facilitating  effect.  But  these  two  means  are  not  significantly 
different.  Moreover,  Pseudonarrative  wihtout  Visual  Deixis  and  with  Structure  marking  does 
not  differ  significantly  from  Reasoning  without  Visual  Deixis  and  without  Structure  Marking, 
wihch  is  the  least  effective  stimulus  in  this  series.  One  possible  explanation  is  that  subjects  find 
spoken  Pseudonarrtive  without  Visual  Deixis  and  with  Structure  Marking  very  strange,  since  it 
is  a  discourse  type  which  is  based  on  spoken  Narrative,  a  very  familiar  and  interactive  discourse 
type  in  which  gesture  may  be  an  integral  and  expected  component.  It  is  interesting  to  note  that 
for  Pseudonarrative,  having  either  Visual  Deixis  or  Structure  Marking  alone  is  less  effective 
than  having  neither. 

5.2  Boolean  Level 

We  now  discuss  the  results  of  experiments  at  the  Boolean  level,  the  level  which  attempts  to 
convey  the  Boolean  functions  of  the  switch  positions  which  the  lights  represent. 

5.2.1  Effect  of  Diagram  Structure 

The  manipulation  at  the  Boolean  level  which  showed  the  strongest  effect  was  varying  the 
diagram  on  the  blackboard.  The  standard  diagram,  used  with  six  of  the  seven  Boolean  stimuli, 
consisted  of  equations  corresponding  directly  to  the  script  spoken  by  the  instructor.  The 
diagram  used  in  the  seventh  version,  shown  in  Figure  12,  is  not  coordinateded  with  the  script, 
since  it  uses  an  "X"  to  indicate  that  a  given  light  is  on  for  a  given  switch  setting,  and  this  use  of 
•X*  is  not  explained  at  all  in  the  discourse,  which  is  about  equations. 


Figure  12:  Uncoordinated  Diagram  Used  with  Boolean  Level  Explanations 

For  an  example  of  how  this  diagram  was  used,  as  the  instructor  says 
C  equals  switch  2 
he  points  to  its  first  column. 
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We  had  expected  that  this  (rather  drastic)  lack  of  integration  would  produce  comprehension 
problems.  In  fact,  the  opposite  was  true.  This  may  be  explained  by  the  fact  that  even  though 
the  diagram  is  not  coordinated  with  the  script,  it  is  in  fact  closer  to  the  form  of  many  of  the 
questions  than  the  equations  are.  Thus,  for  a  question  like 

Can  yon  got  fertilized  water  coneieting  of  the  nixtnre  A,  C.  D  of 
chemicals? 

it  is  possible  to  obtain  the  answer  directly  from  a  single  row  of  the  diagram.  In  constrast,  to 
obtain  an  answer  using  the  equations  would  require  considering  three  separate  equations. 

This  result  indicates  the  need  for  further  research  on  chart  and  diagram  structure.  Although 
there  has  been  some  research,  particularly  on  how  to  assess  the  quality  of  graphs  (see  [Koslyn  et 
al,  83]  for  a  review),  the  field  is  just  beginning.  We  know  that  a  good  icon  can  be  immensely 
valuable,  but  we  do  not  yet  know,  except  in  very  local  instances,  how  to  create  one  for  a 
particular  purpose.  Some  suggestions  for  how  to  advance  the  state  of  the  art  based  on  the 
results  of  the  present  research,  are  given  in  Section  5.5. 

5.2.2  Cognitive  Level  Comparisons 

To  explore  the  effect  of  cognitive  level  on  comprehension,  we  can  compare  the  standard 
Reasoning  stimuli  from  the  two  series,  using  the  total  score  on  the  items  common  to  the  two 
questionnaires  as  the  dependent  measure.  As  predicted,  a  Combinatorial  level  explanation 
produces  far  superior  comprehension  than  the  Boolean  level,  for  this  task.  Actually,  more  than 
this  is  true.  The  intermediate  case,  of  Boolean  level  reasoning  with  a  Combinatorial  level  icon, 
yields  scores  between  those  of  the  pure  Boolean  and  Combinatorial  stimuli;  the  3-IeveI  ANOVA 
showed  a  strong  main  effect  for  explanatory  condition. 

The  Boolean  level  stimulus  appears  to  be  too  complex,  introducing  too  much  difficult  material 
in  a  short  presentation.  Should  an  explanation  at  this  level  be  necessary  for  some  task,  it 
should  probably  be  longer  than  the  one  used  in  this  experiment,  which  was  controlled  to  have 
equal  length  across  stimuli.  Alternatively,  Boolean  explanations  might  be  appropriate  only  for 
learners  who  are  cognitively  already  familiar  with  the  form  of  Boolean  operations.  (The 
domain-specificity  of  cognitive  complexity  ,  for  instance,  suggests  this  hypothesis.) 

This  result  underlines  the  need  to  develop  a  theory  of  cognitive  levels  (as  discussed  in  Section 
2.2).  Although  we  have  found  cognitive  level  to  be  an  important  variable  for  this  task,  this 
result  can  not  be  extended  to  other  content  areas  until  a  general  theory  of  cognitive  level  is 
available. 


5.2.3  Cross-Level  Discourse  Structure  Permutations 

Conditions  B4,  B5,  and  B6  can  be  used  to  test  the  effect  of  deformations  of  discourse  structure 
at  increasing  levels  of  structure.  Stimulus  B4  changes  focus  within  and  between  adjacent 
sentences.  That  is,  one  clause  (or  sentence)  may  have  as  its  focus  the  lights,  while  the  next  has 
has  its  focus  the  switches  of  the  logic  box.  B5  has  sentences  or  main  clauses  in  scrambled  order 
within  a  discourse  subunit,  but  preserves  the  order  of  subunits.  B6  has  its  discourse  subunits  in 
scrambled  order,  but  scrambled  so  that  no  concepts  are  used  before  they  are  introduced.  Mean 
scores  show  that  the  greater  the  unit  of  structure  over  which  the  deformations  are  performed, 
the  harder  the  resulting  stimulus  is  to  comprehend.  That  is,  change  of  focus  at  the  sentential 
level  produces  better  comprehension  than  scrambled  sentences  within  a  discourse  unit,  which  in 
turn  produces  better  comprehension  than  scrambled  discourse  sub-units.  This  is  predicted  by 
our  theory  of  semiotic  levels.  (The  difference  between  the  first  and  third,  is  significant,  but  the 
differences  from  the  second  are  not.) 

An  unexplained  finding  is  that  the  three  stimuli  involving  discourse  structure  permutations  have 
mean  scores  equal  to  or  better  than  the  reference  condition,  that  is,  the  stimulus  which  is  in  the 
normal  order  at  all  semiotic  levels.  At  present,  we  have  no  explanation  for  this  result;  further 
research  is  required  to  clarify  it.  As  previously  mentioned,  the  first  piece  of  research  would  be 
to  repeat  the  administration  of  the  reference  condition,  to  determine  whether  some  anomaly 
either  in  the  subjects  or  the  conditions  of  administration  led  to  unexpectedly  poor  results. 

(Such  replication  would  be  relatively  easy,  since  the  videotaped  stimulus  is  immediately 
useable.) 

5.2.4  Effect  of  Summaries 

The  facilitating  effect  of  summaries  was  studied  in  the  Boolean  Series.  The  means  for  the 
stimuli  which  had  filler  material  in  place  of  the  initial  and  the  final  summary  were  lower  than 
the  means  for  the  stimulus  having  both  initial  and  final  summaries;  this  is  the  expected  result. 
The  difference  between  the  means  for  the  initial  and  reference  cases  was  marginally  significant. 
We  conclude  that  the  experiment  should  be  repeated,  with  a  2-way  fully-crossed  design. 

It  is  part  of  general  folk  wisdom  about  effective  speaking  and  writing  that  summaries  are 
important.  However,  to  our  knowledge,  previous  research  has  not  considered  summaries  as  part 
of  the  discourse  structure  of  the  experimental  stimuli.  Rather,  in  many  experiments, 
elaborations,  priming  questions  or  other  materials  were  presented  before  the  text  proper  began, 
sometimes  with  a  considerable  time  delay.  This  means  that  they  comprise  a  new  discourse  unit, 
not  an  integral  part  of  the  discourse  unit  being  studied.  Elaborations  are  similar  to  summaries 


in  that  they  are  intended  to  give  the  readers  or  hearers  information  which  helps  them 
understand  the  text  [Reder  80].  However,  summaries  are  defined  both  in  terms  of  the  type  of 
information  they  contain  and  as  a  structural  unit  within  a  given  discourse  type,  while 
elaborations  are  defined  only  in  terms  of  the  the  type  of  information  they  contain. 

The  study  of  the  effects  on  comprehension  of  the  component  parts  of  a  text  is  in  itself  very 
important,  since  it  directly  impacts  the  design  of  instructional  materials.  In  spite  of  the 
difference  in  approach,  there  are  some  similarities  between  this  body  of  research  and  the  results 
of  the  present  study.  For  example,  the  literature  on  elaborations  shows  effects  similar  to  the 
the  result  that  initial  summaries  are  more  effective  than  final  summaries.  This  suggests  that 
initial  summaries  may  in  fact  have  a  priming  effect  in  suggesting  to  subjects  what  to  listen  for. 

5.2.5  Results  Counter  to  Hypotheses 

There  are  several  results  in  the  Boolean  Series  that  were  contrary  to  our  a  priori  hypotheses, 
and  that  still  seem  somewhat  puzzling. 

1.  Boolean  level  explanation  with  Combinatorial  diagram  was  the  most  effective  stimulus. 

We  had  originally  thought  that  the  inconsistency  between  the  visual  icon  and  the 
explanation  would  be  hard  to  understand.  But  in  fact,  the  icon  was  very  close  to  what 
the  questionnaire  actually  asked;  one  might  well  say  "a  picture  is  worth  a  thousand 
words,"  at  least  if  it  is  an  icon  that  is  well-suited  to  the  task. 

2.  We  had  postulated  that  the  reference  stimulus,  Boolean  explanation  with  Boolean  level 
diagram,  would  be  the  most  effective.  However,  two  of  the  boundary  violating  stimuli, 
namely  Focus  Inconsistency  and  Discourse  Unit  Boundary  Violations,  seemed  to  actually 
improve  comprehension  over  the  reference  stimulus,  (although  this  difference  was  not 
significant).  This  result  is  counter-intuitive,  and  indicates  a  need  for  further  research,  in 
particular,  to  repeat  the  administration  of  the  reference  stimulus.  It  was,  in  fact,  the  first 
stimulus  administered  by  an  inexperienced  researcher,  and  it  seems  possible  that  the 
conditions  of  administration  may  be  responsible  for  the  puzzling  results. 

5.3  Suggestions  for  Further  Research 

The  results  reported  here,  although  very  encouraging  for  an  initial  exploration  into  new 
territory,  are  in  many  ways  incomplete.  In  particular,  the  hyptheses  discussed  in  the  current 
report  form  a  relatively  small  subset  of  the  much  larger  set  generated  by  our  conceptual 
framework,  many  others  of  which  also  appear  to  be  promising  candidates  for  empirical  research. 
In  addition,  we  have  identified,  and  indicated  earlier  in  this  report,  a  number  of  experiments 
that  should  be  refined  and  repeated. 
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Taking  a  somewhat  broader  perspective,  we  believe  that  the  initial  (partial)  success  of  this 
program  suggests  a  somewhat  radical  approach  to  computer  graphics  paralleling  the  structures 
found  in  successful  natural  multimedia  interaction:  a  hierarchically  structured  icon  having  a 
rich  system  of  indices  into  and  out  of  the  task  domain,  with  clearly  marked  boundaries.  In  our 
experiments,  the  diagram  used  in  the  Combinatorial  series  is  such  an  icon;  as  noted  above,  its 
structure  corresponds  closely  to  that  of  the  tasks  required  by  the  test  questions.  I  would  be 
possible  to  makes  the  rows,  columns,  and  cells  in  such  an  icon  "mouse  sensitive,"  so  that 
interesting  classes  of  tasks  could  be  performed  simply  with  graphical  manipulations.  The 
success  of  the  "Visicalc"  class  of  programs  can  be  understood  in  this  light.  The  currently 
popular  use  of  icons  in  menus  is  very  crude  by  comparison  with  such  an  approach,  which  we 
believe  could  be  very  valuable  to  explore  further. 

5.4  Recommendations  for  Practical  Multimedia  Instruction  Production 

In  addition  to  suggesting  various  bits  of  further  research,  our  results  can  be  pulled  together  to 
suggest  a  set  of  practical  guidelines  for  producing  multimedia  instructional  packages: 

•  Carefully  specify  the  task; 

•  Construct  a  "discourse  plan"  which  details  a  correspondence  between  the  task  structure 
and  the  discourse  structure; 

•  Provide  a  rich  and  relevant  visual  icon  for  the  task; 

•  In  the  design  of  a  visual  icon,  if  tradeoffs  are  necessary,  model  the  task,  not  the  talk; 

•  Use  pseudonarrative,  if  possible; 

•  Mark  discourse  boundaries  clearly  and  explicitly; 

•  Provide  initial  and  final  summaries; 

•  Chose  the  appropriate  cognitive  level  (even  though  cognitive  level  is  a  poorly  defined 
theoretical  concept,  it  can  be  a  useful  guideline  in  practice). 
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I.  Semiotics 

This  appendix  formulates  precise  notions  of  "sign  system*  (inspired  by  the  thought  of  Charles 
Sanders  Peirce)  and  of  "semiotic  morphism,"  which  is  a  translation  from  signs  in  one  system  to 
signs  (i.e. ,  representations)  in  another.  It  also  considers  what  makes  some  translations  better 
than  others.  This  supports  a  generative  approach,  in  which  optimal  mulitmedia  are  generated 
logical  representations  (which  Peirce  would  have  called  ■  interpretan ts").  A  precise  and  formal 
approach  is  needed  as  a  basis  for  computer  programs  to  embody  our  theories,  and  is  also  helpful 
in  particular  by  suggesting  hypotheses  to  be  tested. 

These  ideas  should  be  applicable  to  many  aspects  of  instruction  as  well  as  to  other  areas  of 
communication.  Some  instances  include  generating  appropriate  explanations,  determining  good 
representations  ("icons")  for  computer  graphics,  measuring  the  quality  of  analogies,  and 
choosing  good  names  for  files  in  a  directory.  There  is  a  ver>  lrge  literature  that  is  relevant  to 
problems  like  these.  However,  all  studies  that  we  know  are  restricted  to  a  particular  sign 
system  (such  as  natural  language)  and/or  a  particular  semantic  domain,  or  else  lack  the 
precision  of  the  theoretical  model  that  we  will  present.  Some  recent  research,  however,  is  fairly 
close  to  ours  in  spirit.  In  particular,  the  problems  of  analogs  and  File  names  mentioned  above 
have  been  studied  by  (Gentner  83]  and  (Carroll  82]  respectively,  who  reach  conclusions 
compatible  with  ours;  specifically,  they  emphasize  the  importance  of  structure  as  opposed  to 
content.  However,  their  theoretical  frameworks  lack  the  hierarchical  levels  and  constructor 
functions  which  enrich  ours. 

A  good  deal  of  [Peirce  65]’s  work  on  the  nature  of  signs  is  directly  relevant  to  the  study  of 
multimedia  instruction.  For  example,  Peirce  divided  signs  into  icons,  symbols  and  indices, 
according  to  the  manner  in  which  they  signify.  Peirce  defines  an  Icon  as  a  "sign  which  refers 
to  the  Object  that  it  denotes  merely  by  virtue  of  characters  of  its  own,"  "such  as  lead-pencil 
streak  as  representing  a  geometrical  line."  In  contrast,  a  sign  x  is  an  index  for  an  object  y  if  x 
and  y  are  regularly  connected,  in  the  sense  "that  always  or  usually  when  there  is  an  x,  there  is 
also  a  y  in  some  more  or  less  exactly  specifiable  spatiotemporal  relation  to  the  x  in 
question"  [Alston  67].  "Such,  for  instance,  is  a  piece  of  mould  with  a  bullet-hole  in  it  as  sign  of 
a  shot"  [Peirce  65].  In  this  example,  the  spatio-temporal  relation  is  a  causal  one,  and  hence  is 
quite  generally  applicable.  However,  there  are  also  indices  which  function  only  in  specified  and 
restricted  spatio-temporal  contexts;  examples  include  proper  names,  social  security  numbers  and 
football  player  numbers.  Finally,  Peirce  defines  a  symbol  as  a  "sign  which  is  constituted  a 
sign  merely  or  mainly  by  the  fact  that  it  is  used  and  understood  as  such." 
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The  notion  of  sign  system  used  in  this  work  (formally  summarized  in  Definition  1  below)  is 
inspired  by  Peirce’s  study  of  semiotics;  it  will  be  seen  that  most  of  our  discussion  concerns  the 
structure  of  complex  signs.  Some  formalization  of  the  structure  of  a  system  of  related  signs  is 
needed  in  order  to  study  what  makes  one  representation  of  a  given  sign  better  than  another. 
This  is  because  it  is  necessary  to  consider  how  to  represent  related  (but  significantly  different) 
signs  to  avoid  confusion  and  ambiguity;  it  is  also  necessary  to  consider  what  attributes  of  signs 
should  be  given  priority  in  constructing  their  representations. 

In  all  but  the  simplest  sign  systems,  individual  signs  are  organized  into  complex  signs;  for 
example,  sentences  are  made  of  words.  This  is  a  fundamental  strategy  for  rendering  the 
complexity  of  non-trivial  communication  more  manageable.  One  may  iterate  this  strategy,  by 
regarding  complex  signs  at  one  level  of  analysis  as  individual  signs  at  a  higher  level,  and  then 
forming  complex  signs  from  these  as  well,  which  leads  to  a  multilevel  hierarchy  of  sign 
structure.  For  example,  linguistics  recognizes  the  following  levels:  phonological  (the  sounds  of  a 
given  language);  morphological  (the  smallest  repeated  compounds  of  phonemes  with  a  stable 
meaning);  lexical  (words);  syntactic  (phrases  and  sentences);  and  discourse  (multisentential 
units).  It  is  important  to  note  that  this  is  a  "whole/part*  hierarchy,  in  which  items  at  each 
level  are  composed  from  components  from  the  next  lower  level. 

The  whole/part  hierarchical  organization  of  complex  signs  requires  that  signs  be  considered  not 
only  individually  but  in  their  context.  The  immediate  context  of  a  sign  consists  of  those  other 
signs  that  surround  it,  in  space  and/or  time,  and  that  together  with  it  form  a  complex  sign  at 
the  next  higher  level.  In  numerous  linguistic  studies,  it  has  been  found  that  the  context  and 
speaker  of  a  given  sentence  in  a  story  are  at  least  as  important  for  determining  its  meaning  as 
are  the  words  that  comprise  it.  (For  an  extreme  example,  the  sentence  "Yes"  can  mean  almost 
anything  when  given  an  appropriate  context.)  Generalizing  this,  we  may  say  that  it  is  more 
useful  to  view  meaning  as  being  produced  'top-down*  rather  than  "bottom-up.*  (An  example 
of  the  utilitiy  of  this  view  is  found  in  artificial  intelligence  research,  where  contextual  cues  have 
been  found  to  be  essential  in  recognizing  and  disambiguating  signs;  this  has  particularly  been 
true  for  speech  understanding  and  machine  vision  projects.) 

An  important  aspect  is  the  particular  medium  in  which  a  sign  is  expressed.  Some  media  are 
sensory,  e.g.,  visual,  auditory,  and  kinesic  representations.  In  addition,  there  are  many 
important  examples  of  mental  sign  activity,  such  as  inferring  a  proposition  from  one  or  more 
others.  Signs  may  involve  more  than  one  medium;  for  example,  a  telephone  conversation 
involves  the  auditory  medium,  while  a  television  program  involves  an  audio-visual  mix.  The 


signs  of  any  given  system  have  some  particular  mixture  of  media  in  which  they  are  expressed. 
For  our  purposes,  it  is  more  convenient  to  regard  a  mixture  of  media  as  a  single  mixed  medium, 
called  the  medium  of  the  sign  system;  we  may  also  call  such  a  sign  system  mixed. 

A  very  large  number  of  different  signs  may  be  possible  in  a  given  mixed  medium;  and  it  may 
also  be  possible  to  organize  these  signs  (or  subsets  of  them)  in  a  wide  variety  of  different  ways. 
Within  a  given  mixed  medium,  a  specific  choice  of  signs  and  way  of  organizing  them  may 
characterize  a  particular  sign  system  (of  course  other  factors  may  also  be  involved.)  Thus,  a 
sign  that  is  meaningful  in  one  sign  system  may  not  be  in  another,  even  though  they  involve  the 
same  medium.  For  example,  different  alphabets  (such  as  Roman,  Greek  and  Cyrillic)  involve 
different  letters,  although  a  given  form  for  a  letter  may  be  used  in  more  than  one  alphabet. 

We  now  consider  the  structure  of  signs  at  a  given  level  of  the  whole/part  hierarchy  of  levels. 
Some  of  this  discussion  may  be  familiar  from  formalisms  used  in  linguistics,  but  our  purpose 
here  is  to  develop  an  approach  that  is  applicable  to  any  sign  system  whatever.  It  is  important 
to  note  that  signs  at  level  n  are  constructed  from  signs  at  level  n-1  (and  other  signs  at  level  n 
that  are  already  so  constructed).  A  given  sign  system  admits  only  a  certain  limited  number  of 
ways  to  put  parts  together  at  a  given  level  n;  we  will  refer  to  these  as  its  constructors  at  level 
n.  In  general,  there  is  a  classification  scheme  for  the  signs  at  a  given  level  (e.g.,  the  parts  of 
speech  are  such  a  scheme  for  the  syntactic  level  of  a  natural  language  system),  and  the 
constructors  at  that  level  can  be  seen  as  rules  for  combining  signs  of  these  various  classes  to 
get  a  new  sign  of  another  class.  A  familiar  special  case  is  that  of  context-free  grammars. 

Three  additions  to  this  basic  scheme  seem  useful;  there  may  well  be  others  that  we  have  not 
discovered.  First,  a  given  constructor  (or  rule)  may  have,  in  addition  to  its  formal  arguments, 
some  number  of  parameters.  For  example,  consider  a  graphic  sign,  say  a  cat,  to  be  displayed 
on  a  graphics  terminal;  its  parameters  might  be  its  size,  and  the  location  of  its  upper-lefthand 
corner;  depending  on  their  values,  the  cat  may  have  a  different  size  and  location,  but  still  be 
recognizably  the  same  cat. 

The  second  addition  is  that  there  may  be  a  priority  ordering  on  constructor  functions. 

Under  such  an  ordering,  there  may  be  a  primary  constructor,  which  has  greater  priority 
than  any  other  constructor;  there  may  also  be  one  or  more  secondary  constructors,  each 
having  less  priority  than  the  primary  constructor  and  greater  priority  than  any  non-primary 
and  non-secondary  constructor,  with  none  of  these  having  priority  over  any  other;  similarly, 
there  may  be  one  or  more  tertiary  constructors,  and  so  on.  This  formulation  yields  a  partial 


ordering  rather  than  a  total  ordering,  since  given  two  distinct  constructors  rl  and  r2,  it  is  not 
necessary  that  either  one  has  priority  over  the  other.  For  example,  the  reasoning  discourse 
type  [Goguen,  Weiner  &  Linde  83]  has  a  primary  constructor,  AND,  which  conjoins  a  number  of 
reasons  for  the  same  statement4.  Unfortunately,  there  has  not  yet  been  any  experimental  work 
on  this  subject,  which  would  be  an  interesting  extension  of  the  work  done  for  the  present 
project. 

The  third  addition  is  that  of  context  conditions^  on  rules.  These  are  conditions  (i.e., 
predicates)  that  limit  the  applicability  of  a  rule  to  certain  particular  contexts,  namely  those 
where  the  predicate  is  true.  Context  conditions  often  express  constraints  that  arise  as  a  result 
of  structure  at  higher  levels  of  the  hierarchy,  and  are  a  feature  of  many  recent  grammatical 
formalisms  [Kaplan  &  Bresnan  82],  [Wasow  et  al.  82]. 

Many  aspects  of  this  approach  to  the  structure  of  signs  are  present  or  implicit  in  Peirce,  but  as 
far  as  we  can  tell,  these  considerations  were  never  explicitly  assembled  into  a  single  precise 
definition.  Our  purpose  in  doing  so  is  to  make  as  explicit  as  possible  what  is  involved  in 
constructing  (or  in  attempting  to  construct)  optimal  explanations  or  other  representations  of 
instructional  material. 

We  emphasize  the  structure  of  the  system  of  related  signs.  The  basic  insight  underlying  this 
definition  is  that  semiotic  events  are  not  isolated  phenomena,  but  rather  occur  in  systems:  there 
are  common  rules  relating  to  the  recognition,  construction,  denotation  and  interpretation  of 
such  a  collection  of  signs.  Moreover,  semiotics  as  a  subject  is  (or  should  be)  more  concerned 
with  such  rules  than  with  the  comparative  study  of  individual  signs  and  the  settings  in  which 
they  are  found.  (This  distinction  is  like  the  distinction  between  descriptive  biology  and  modern 
biology  that  is  based  on  biochemistry  and  molecular  biology.)  It  should  be  noted  that  this 
definition,  like  the  others  in  this  section,  is  set-theoretic  rather  than  axiomatic.  Thus,  it  avoids 
issues  of  completeness  and  restrictions  to  denumerable  sets  of  signs,  properties,  functions,  etc. 
Definition  1:  A  sign  system  is  a  class  of  entities,  called  signs,  divided  into  a  set  of  levels 
(numbered  1  to  N  and  not  necessarily  disjoint),  such  that  signs  at  level  n+I  (for  l<n<N)  are 

4 

The  way  to  determine  the  primary  constructor  for  a  given  discourse  type  is  to  determine  the  •default* 
constructor  for  that  type,  i.e.,  the  constructor  that  is  assumed  to  hold  when  in  fact  there  is  no  explicit  marking  in 
the  text.  For  narrative,  if  one  sentence  follows  another,  it  is  assumed  that  they  are  connected  by  a  SEQUENCE 
constructor;  this  is  just  the  narrative  presupposition. 

'’This  is  a  limited  and  technical  sense  of  ■context.*  A  broader  sense  has  been  given  earlier;  a  still  broader  sense 
would  take  account  of  pragmatics. 


constructed  from  signs  at  level  n  (and  other  signs  at  level  n+1)  by  use  of  a  fixed  set  of 
constructor  functions  (these  may  also  have  parameters  and  context  conditions).  In  addition, 
there  may  be  a  priority  (partial)  ordering  on  these  constructors  at  each  level;  and  for  each 
level,  there  may  be  predicates,  relations  and  functions  expressing  properties  of  signs  at  that 
level.  Finally,  a  sign  system  has  an  associated  medium  for  its  signs,  and  is  mixed  if  that 
medium  is  mixed.  [) 

We  may  illustrate  the  concepts  in  this  definition  with  the  system  of  spoken  English.  The 
underlying  medium  is  sound,  that  is,  acoustic  signals.  The  signs  are  classed  into  the  usual  levels 
of  spoken  English,  phones,  phonemes,  morphemes,  words,  phrases,  sentences,  and  discourse 
units.  Constructors  at  the  sentence  level  are  given  by  rules,  as  studied  in  linguistics. 

Artificial  systems  often  exhibit  the  structures  in  this  definition  in  a  very  natural  way.  For 
example,  let  us  consider  a  simple  line-oriented  editor  for  a  standard  24  line  by  80  character 
screen.  The  lowest  hierarchical  level  is  that  of  characters,  the  second  that  of  lines,  the  third 
that  of  screcnfulls,  and  the  last  that  of  sequences  of  screens;  thus,  signs  at  the  second  level 
consist  of  strings  of  80  or  fewer  characters,  and  signs  at  the  third  level  consist  of  strings  of  24  or 
fewer  lines.  We  can  see  this  simple  system  as  having  just  one  constructor  at  each  level  greater 
than  1,  namely  STRING-OF(al,...,aJ,  J)  with  parameter  J,  which  "strings  together"  J  signs 
from  the  next  lower  level.  The  second  level  has  the  context  condition  0<J<80,  and  the  third 
0<  J<24.  Since  there  is  at  most  one  constructor  at  each  level,  the  priority  ordering  is  trivial. 
However,  there  are  some  interesting  predicates  and  functions,  such  as  the  LINE-LENGTH 
function  for  lines  and  the  ALPHANUMERIC  predicate  for  characters. 

We  can  also  illustrate  this  definition  in  the  domain  of  computer  graphics.  Example  signs 
include  lines,  characters,  circles  and  squares,  Levels  might  consist  of  pixels  (individual  "dots" 
on  the  screen),  lines,  simple  figures,  and  windows  (consisting  of  arbitrary  "scenes,"  which  are 
collections  of  signs  at  lower  levels,  plus  other  windows);  each  sign  at  each  level  must  also  have 
associated  attributes  for  location  on  the  screen,  and  size;  there  may  also  be  attributes  for  color 
and  intensity.  An  especially  interesting  constructor  here  is  WINDOW,  which  can  recursively 
encapsulate  signs  from  all  levels. 

Let  us  now  focus  on  our  primary  concern,  the  translation  of  signs  in  one  system  to  signs  in 
another  system.  It  is  our  intention  to  provide  the  theoretical  background  for  a  general  theory 
of  the  construction  and  interpretation  of  signs.  For  example,  generating  an  optimal  (or 
reasonably  good)  explanation,  generating  appropriate  graphical  icons,  choosing  a  good  file 


name,  choosing  a  good  analogy,  and  understanding  texts  and/or  graphics,  can  all  be  seen  as 
problems  of  translating  signs  from  one  sign  system  into  another.  Notice  that  the  problem  of 
choosing  an  optimal  mix  of  media  also  falls  in  this  framework,  since  we  can  regard  the  signs  in 
a  given  media  mix  as  forming  a  subsystem  of  the  total  sign  system  within  which  we  must  choose 
representations.  We  address  questions  about  the  nature  of  translations  between  sign  systems, 
and  the  reasons  for  preferring  one  translation  to  another,  by  studying  maps  from  signs  in  one 
system  to  representation  signs  in  another  system. 

Definition  2:  Let  51  and  52  be  sign  systems.  Then  a  (semiotic)  morphism  M:  51  — ►  52, 
from  51  to  52,  consists  of  the  following  partial  functions  (all  denoted  M): 

1.  Signs  of  51  — ►  Signs  of  52, 

2.  Levels  of  5l  — ♦  Levels  of  52, 

3.  Classes  of  51  at  Level  i  — »  Classes  of  52  at  Level  M(i), 

4.  Constructors  of  5l  at  Level  i  —*■  Constructors  of  52  at  Level  M(i),  and 

5.  Property  Predicates  and  Functions  of  51  at  Level  i  — ►  Property  Predicates  and  Functions 
of  52  at  Level  M(i), 

such  that 

1.  if  i<j  (for  l<i,j<N,  where  N  is  the  number  of  levels  of  5l)  then  M(i)<M(j), 

2.  if  r:  <cl>...<ck>  -*  <c>  is  a  constructor  at  level  i  of  51,  then 

M(r):  M(<cl>)...M(<ck>)  -*•  M(<c>)  is  a  constructor  at  level  M(i)  of  52  (if  it  is 
defined),  and 

3.  if  p:  <cl>...<ck>  is  a  predicate  at  level  i  of  51,  then  M(p):  M(<cl>)...M(<ck>)  is  a 
predicate  at  level  M(i)  of  S2  (if  it  is  defined). 

0 

Such  a  mapping  may  or  may  not  preserve  the  structure  of  a  sign  system.  The  degree  to  which 
it  does  so  affects  the  quality  of  its  representations.  A  simple  example  of  a  semiotic  morphism 
from  the  domain  of  our  experimental  work  is  the  correspondance  between  the  physical  order  of 
lights  on  the  experimental  box,  and  the  order  in  which  clauses  are  given  to  describe  the  lights 
(narrative  order). 

Our  fuller  exposition  of  this  material  in  [Goguen  &  Linde  84)  includes  detailed  definitions  of 
what  it  means  for  a  semiotic  morphism  M  to  be  level  preserving,  constructor  preserving, 
priority  preserving,  property  preserving,  and  structure  preserving.  It  is  important  to 
notice  that  semiotic  morphisms  need  not  be  totally  defined;  that  is,  each  of  the  functions 
denoted  M  can  be  undefined  on  some  of  what  is  in  51.  For  example,  there  need  not  be  any 
representation  in  52  for  some  signs  in  51;  in  particular,  some  components  of  M  could  even  be 
totally  undefined,  i.e.,  the  empty  function. 


36 


It  seems  clear  that  a  structure  preserving  semiotic  morphism  M:  Si— »S2  will  faithfully  represent 
all  of  the  semiotic  structure  of  Si  in  terms  of  that  available  in  52.  This  might  seem  to  be 
desirable  for  an  optimal  representation;  however,  if  the  resulting  structures  in  Si  are  too 
complex,  then  they  may  be  hard  for  human  beings  to  understand.  For  example,  if  Si  consists 
of  parse  trees  for  English  sentences  and  S2  consists  of  the  usual  “printed  page"  text  format, 
then  it  is  possible  to  translate  all  the  syntactic  information  that  is  available  in  Si  into  structures 
in  S 2  with  so-called  “phrase  structure"  notation,  which  uses  brackets  to  delimit  phrases  and 
uses  subscripts  on  brackets  to  indicate  the  class  of  phrase  is  involved.  For  example, 

([[[tMoet  [light]NWHonlprept[tbe]Det[left]NlNp]ppWHcomes]vIOD)partlvplsentence 
This  may  be  useful  for  some  purposes,  but  it  is  clearly  not  optimal  for  others.  There  seems  to 

be  a  trade-off  between  the  degree  of  structure  preservation  and  the  degree  of  complexity  of  the 
resulting  representations. 

We  now  turn  to  the  complex  issue  of  determining  whether  one  representation  (i.e.,  semiotic 
morphism)  is  better  than  another.  One  evident  consideration  is  whether  it  preserves  more 
structure. 

Definition  3:  Let  M'  and  M  be  morphisms  from  sign  system  Si  to  sign  system  52.  Then  M' 
preserves  as  least  as  much  structure  as  M  does,  provided  that: 

1.  if  M  preserves  a  sign  e  at  level  i,  then  so  does  ]Vf; 

2.  if  M  preserves  a  constructor  r  at  signs  el,...,ek,  then  so  does  M'; 

3.  if  M  preserves  a  priority  r>r/  then  so  does  M*;  and 

4.  if  M  preserves  a  property  p,  then  so  does  M*.  [) 

The  interesting  problems  arise  comparing  morphisms  M  and  M*  when  neither  preserves  strictly 
more  structure  than  the  other,  or  when  one  preserves  more  structure  but  also  produces  more 
complex  representations.  For  example,  M  might  preserve  more  levels  than  M',  whereas  M' 
preserves  more  properties  than  M.  It  is  for  such  cases  that  experimental  work  is  especially 
important.  Our  framework  suggests  that  preserving  levels  is  more  important  than  preserving 
priorities,  which  is  more  important  than  preserving  properties.  In  short,  structure  is  more 
important  than  content. 

Our  experimental  results  suggest  that  the  higher  the  level  of  structure,  the  more  important  it  is 
to  preserve  that  structure.  This  is  an  exciting,  and  somewhat  unexpected,  finding;  we  had 
originally  expected  a  more  complex  ordering  on  levels,  and  had  posited  a  basic  level  that  was 
not  necessarily  the  highest  level,  by  analogy  with  work  by  Rosch  and  her  students,  explicating 
the  notion  of  “basic  level"  for  lexical  concepts  (e.g.,  “bird")  [Rosch  73,  Rosch  75].  In  the  case 


of  natural  language,  for  the  last  twenty  five  years  it  has  been  supposed  that  the  sentential  level 
is  basic,  although  more  recent  research  is  inclined  to  regard  the  discourse  level  as  at  least  as 
important. 

The  body  of  this  report  described  the  results  of  testing  some  experimental  hypotheses  related  to 
these  issues.  We  hope  that  this  combination  of  theoretical  and  experimental  research  will 
eventually  make  it  possible  to  effectively  determine  of  a  given  representation  whether  or  not  it 
is  adequate  to  the  task  in  hand.  More  optimistically,  given  sign  systems  51  and  52,  where  51 
contains  abstract  forms  of  the  information  to  be  conveyed,  we  would  like  to  generate 
(preferably  automatically)  a  semiotic  morphism  M  from  51  to  52  that  will  give  adequate 
representations  in  52  for  signs  from  51.  For  example,  51  might  contain  instructions  for 
repairing  some  piece  of  equipment,  and  52  might  be  a  color  graphics  terminal  with  a  speech 
chip  and  audio  output.  The  problem  is  then  to  generate  instructional  material  that  effectively 
utilizes  the  capabilities  of  that  particular  terminal. 
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IT.  Scripts  for  Experimental  Stimuli 

Each  stimulus  begins  with  exactly  the  same  Task  Description,  which  is  given  here  once  and  for 
all,  rather  than  repeated  fifteen  times: 

Today  we’re  going  to  learn  how  to  operate  this  device,  which  is  a  fertilizer  controller 
for  an  irrigation  system.  The  irrigation  system  delivers  different  mixtures  of  water 
and  chemicals  to  the  fields,  and  this  device  controls  which  chemicals  go  into  the 
water.  Now  your  job  as  controller  is  to  set  up  the  various  mixtures  that  your 
supervisor  tells  you  are  needed.  And  your  job  here  today  is  to  learn  how  to  use  this 
controller.  There  will  be  a  test  after  the  explanation,  so  please  listen  as  carefully  as 
you  can. 


II.l  Combinatorial  Series  Stimuli 

This  section  presents  the  scripts  used  to  produce  the  experimental  stimuli  used  in  the 
Combinatorial  Series  of  experiments.  The  diagram  shown  in  Figure  12  was  used  with  all  stimuli 
in  this  series. 


n.1.1.  Cl  Script 

This  is  the  case  of  Pseudonarrative  with  Visual  Deixis  and  without  Structure  Marking. 
Initial  Summary: 

This  fertilizer  controller  has  two  switches  at  the  bottom,  and  they’re  numbered  1  and 
2.  And  we’ll  see  that  there  are  four  ways  to  set  these  two  switches.  Also  this 
controller  has  four  lights  here,  they’re  labelled  A,  B,  C  and  D,  and  they  show  which 
chemicals  are  being  added  to  the  water.  Now  the  setting  of  the  switches  controls 
which  chemicals  are  added,  and  each  chemical  is  represented  by  a  light,  so  if  I  move 
these  switches  around,  you  see  that  different  lights  turn  on  and  off.  And  because  you 
can  get  four  different  possible  settings  of  the  switches,  you  can  also  get  four  different 
mixtures  of  chemicals. 

We  have  this  controller  with  two  switches  and  four  lights  and  we  have  to  learn  how 
to  operate  it.  Let’s  make  sure  that  there  really  are  just  four  possible  ways  to  set 
these  two  switches.  We’ll  start  with  the  first  setting,  which  has  both  switches  up. 
Then  we  go  to  the  second  setting,  which  has  both  switches  down.  Then  we  go  to  the 
third  setting,  which  has  the  first  switch  up  and  the  second  switch  down.  Finally,  we 
go  to  the  fourth  setting,  which  has  the  first  switch  down  and  the  second  switch  up. 
And  that’s  all  the  settings  that  there  are.  This  fertilizer  controller  can  only  select 
among  four  mixtures  of  chemicals  to  add  to  the  water. 

Each  of  these  switch  settings  has  its  own  pattern  of  lights,  that  is,  some  of  the  lights 
are  on  and  others  are  off  when  the  switches  are  set  that  way.  Let’s  start  with  the 
setting  where  we  put  both  switches  up.  That  gives  us  lights  A,  C,  and  D.  Next,  let’s 
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put  both  switches  down.  That  gives  us  lights  A,  B,  and  D.  Then  we  can  put  switch  1 
up  and  switch  2  down  and  that  gives  us  no  lights  on  at  all.  Finally  we  can  put  switch 
1  down  and  switch  2  up  and  that  gives  us  the  lights  A  and  C. 

Let’s  use  this  diagram.  The  top  indicates  the  four  lights,  for  the  four  chemicals,  and 
the  left  side  shows  the  four  switch  settings.  So  first  we  have  1  up  and  2  up  and  we 
get  lights  A,  C,  and  D.  Next  we  go  to  both  down  and  that  gives  us  A,  B  and  D.  Then 
we  go  to  1  up  and  2  down  and  we  get  no  lights  at  all.  And  finally  we  go  to  1  down 
and  2  up  and  that  gives  us  A  and  C. 

Suppose  that  your  supervisor  wants  you  to  get  a  particular  mixture  of  chemicals.  For 
example,  maybe  she  wants  to  get  the  mixture  A,  C,  D.  Then  all  you  have  to  do  is 
look  at  the  diagram  and  see  which  switch  setting  produces  that  mixture.  So,  the  first 
line  shows  us  that  both  switches  up  produces  the  mixture  she  wants. 

Another  thing  that  the  supervisor  might  want  is  just  any  setting  that  will  give  the 
chemical  indicated  by  light  D.  Looking  at  the  diagram,  we  can  see  that  if  either 
Switch  1  and  Switch  2  are  both  up,  or  else  if  Switch  1  and  Switch  2  are  both  down, 
we  will  get  some  of  the  chemical  indicated  by  light  D. 

Final  Summary: 

The  way  this  controller  works  is  that  the  four  lights  indicate  the  chemicals  and  the 
different  patterns  of  lights  depend  on  the  settings  of  the  two  switches.  So  if  you  want 
to  get  a  particular  mixture  of  chemicals,  you  have  to  know  which  switch  setting  gives 
that  mixture.  And  if  you  want  to  know  how  to  get  a  particular  chemical,  you  have  to 
know  which  switch  settings  give  it. 


n.1.2.  C2  Script 

This  is  the  case  of  Pseudonarrative  without  Visual  Deixis  and  without  Structure  Marking.  The 
text  is  identical  to  that  of  Cl,  but  the  actor  did  not  employ  Visual  Deixis. 


n.1.3.  C3  Script 

This  is  the  case  of  Pseudonarrative  with  Visual  Deixis  and  with  Structure  Marking. 

Initial  Summary: 

So  as  you  can  see,  this  fertilizer  controller  has  two  switches  at  the  bottom,  and 
they’re  numbered  1  and  2.  And  we’ll  see  that  there  are  four  ways  to  set  these  two 
switches.  Also  this  controller  has  four  lights  here,  they’re  labelled  A,  B,  C  and  D,  and 
they  show  which  chemicals  are  being  added  to  the  water.  Now  the  setting  of  the 
switches  controls  which  chemicals  are  added,  and  each  chemical  is  represented  by  a 
light,  so  if  I  move  these  switches  around,  you  see  that  different  lights  turn  on  and  off. 
And  because  you  can  get  four  different  possible  settings  of  the  switches,  you  can  also 
get  four  different  mixtures  of  chemicals. 


So  to  begin,  we  have  this  controller  with  two  switches  and  four  lights  and  we  have  to 
learn  how  to  operate  it.  Now  let’s  make  sure  that  there  really  are  just  four  possible 
ways  to  set  these  two  switches.  Go  through  the  four  positions  on  the  box.  We’ll 
start  with  the  first  setting,  which  has  both  switches  up.  Then  we  go  to  the  second 
setting,  which  has  both  switches  down.  Then  we  go  to  the  third  setting,  which  has 
the  first  switch  up  and  the  second  switch  down.  Finally,  we  go  to  the  fourth  setting, 
which  has  the  first  switch  down  and  the  second  switch  up.  And  that’s  all  the  settings 
that  there  are.  So  this  fertilizer  controller  can  only  select  among  four  mixtures  of 
chemicals  to  add  to  the  water. 

Now  each  of  these  switch  settings  has  its  own  pattern  of  lights,  that  is,  some  of  the 
lights  are  on  and  others  are  off  when  the  switches  are  set  that  way.  Let’s  start  with 
the  setting  where  we  put  both  switches  up.  That  gives  us  lights  A,  C,  and  D.  Next, 
let’s  put  both  switches  down.  That  gives  us  lights  A,  B,  and  D.  Then  we  can  put 
switch  1  up  and  switch  2  down  and  that  gives  us  no  lights  on  at  all.  Finally  we  can 
put  switch  1  down  and  switch  2  up  and  that  gives  us  the  lights  A  and  C. 

Now  let’s  go  over  this  again  and  use  this  diagram.  The  top  indicates  the  four  lights, 
for  the  four  chemicals,  and  the  left  side  shows  the  four  switch  settings.  So  first  we 
have  1  up  and  2  up  and  we  get  lights  A,  C,  and  D.  Next  we  go  to  both  down  and  that 
gives  us  A,  B  and  D.  Then  we  go  to  1  up  and  2  down  and  we  get  no  lights  at  all. 

And  finally  we  go  to  1  down  and  2  up  and  that  gives  us  A  and  C.  And  this  diagram 
shows  us  all  of  this. 

Now  suppose  that  your  supervisor  wants  you  to  get  a  particular  mixture  of  chemicals. 
For  example,  maybe  she  wants  to  get  the  mixture  A,  C,  D.  Then  all  you  have  to  do  is 
look  at  the  diagram  and  see  which  switch  setting  produces  that  mixture.  So,  the  first 
line  shows  us  that  both  switches  up  procduces  the  mixture  she  wants. 

Another  thing  that  the  supervisor  might  want  is  just  any  setting  that  will  give  the 
chemical  indicated  by  light  D.  Looking  at  the  diagram,  we  can  see  that  if  either 
Switch  1  and  Switch  2  are  both  up,  or  else  if  Switch  1  and  Switch  2  are  both  down, 
we  will  get  some  of  the  chemical  indicated  by  light  D. 

Final  Summary: 

So  as  you’ve  seen,  the  way  this  controller  works  is  that  the  four  lights  indicate  the 
chemicals  and  the  different  patterns  of  lights  depend  on  the  settings  of  the  two 
switches.  So  if  you  want  to  get  a  particular  mixture  of  chemicals,  you  have  to  know 
which  switch  setting  gives  that  mixture.  And  if  you  want  to  know  how  to  get  a 
particular  chemical,  you  have  to  know  which  switch  settings  give  it. 
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H.1.4.  C4  Script 

This  is  the  case  of  Pseudonarrative  without  Visual  Deixis  and  with  Structure  Marking.  The 
text  is  identical  to  that  of  C3,  but  the  actor  did  not  use  Visual  Deixis. 

n.1.5.  C5  Script 

This  is  the  case  of  Reasoning  with  Visual  Deixis  and  without  Structure  Marking. 

Initial  Summary: 

This  fertilizer  controller  has  two  switches  at  the  bottom,  and  they’re  numbered  1  and 
2.  And  we’ll  see  that  there  are  four  ways  to  set  these  two  switches.  Also  this 
controller  has  four  lights  up  here,  they’re  labelled  A,  B,  C  and  D,  and  they  show 
which  chemicals  are  being  added  to  the  water.  Now  the  setting  of  the  switches 
controls  which  chemicals  are  added,  and  each  chemical  is  represented  by  a  light,  so  if 
I  move  these  switches  around,  you  see  that  different  lights  turn  on  and  off.  And 
because  you  can  get  four  different  settings  of  the  switches,  you  can  also  get  four 
different  mixtures  of  chemicals. 

Here  is  this  controller  with  two  switches  and  four  lights  and  your  job  is  to  learn  how 
to  operate  it.  First,  let’s  make  sure  that  there  really  are  just  four  ways  to  set  these 
switches.  Each  switch  has  exactly  two  settings,  up  and  down.  So  if  1  is  up,  then  2 
could  be  up  or  2  could  be  down,  and  that’s  two  settings.  Now  Switch  1  could  be 
changed  to  down,  and  then  still  2  could  be  up  or  down.  So  that’s  another  two 
settings,  and  which  makes  four  settings  altogether.  So  this  fertilizer  controller  can 
only  select  among  four  mixtures  of  chemicals  to  add  to  the  water. 

Each  of  these  switch  settings  has  its  own  pattern  of  lights,  that  is,  some  of  the  lights 
are  on  and  others  are  off  when  the  switches  are  set  that  way.  For  the  first  setting, 
where  both  switches  are  up,  we  have  lights  A,  C  and  D  on.  And  for  the  second 
setting,  where  both  switches  are  down,  lights  A,  B,  and  D  are  on.  For  the  third 
setting,  with  switch  1  up  and  switch  2  down,  we  get  no  lights  on  at  all.  And  for  the 
last  setting,  with  1  down  and  2  up,  we  see  that  only  lights  A  and  C  are  on.  That’s  all 
four  cases. 

This  diagram  summarizes  everything  that  we  said  before.  The  top  indicates  the  four 
lights,  for  the  four  chemicals,  and  the  left  side  shows  the  four  different  switch 
settings.  So  first  both  1  and  2  are  up,  and  that  has  A,  C,  and  D  on.  Next,  both  1 
and  2  are  down,  and  that  has  A  B  and  D  on.  Then  1  is  up  and  2  is  down,  and  that 
has  no  lights  on  at  all.  And  finally,  1  is  down  and  2  is  up,  and  that  has  A,  and  C  on. 

Now  suppose  that  your  supervisor  wants  you  to  get  a  particular  mixture  of  chemicals. 
For  example,  maybe  she  wants  to  get  the  mixture  A,  C,  D.  Then  all  you  have  to  do  is 
look  at  the  diagram  and  see  which  switch  setting  gives  that  mixture.  So,  the  first  line 
shows  us  that  both  switches  up  gives  the  mixture  she  wants. 
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Another  thing  that  the  supervisor  might  want  is  just  any  setting  that  will  give  the 
chemical  indicated  by  light  D.  Looking  at  the  diagram,  we  can  see  that  if  either 
Switch  1  and  Switch  2  are  both  up,  or  else  if  Switch  1  and  Switch  2  are  both  down, 
we  get  some  of  the  chemical  indicated  by  light  D. 

Final  Summary: 

The  way  this  controller  works  is  that  the  four  lights  depend  on  the  settings  of  the  two 
switches,  and  the  different  patterns  of  lights  depend  on  the  settings  of  the  switches. 

So  if  you  want  to  get  a  particular  mixture  of  chemicals,  you  have  to  know  which 
switch  settings  give  it. 

n.1.6.  C6  Script 

This  is  the  case  of  Reasoning  without  Visual  Deixis  and  without  Structure  Marking.  The  text  is 
identical  to  that  of  C5,  but  the  actor  does  not  use  Visual  Deixis. 

n.1.7.  C7  Script 

This  is  the  case  of  Reasoning  with  Visual  Deixis  and  with  Structure  Marking. 

Initial  Summary : 

So  as  you  can  see,  this  fertilizer  controller  has  two  switches  at  the  bottom,  and 
they’re  numbered  1  and  2.  And  we’ll  see  that  there  are  four  ways  to  set  these  two 
switches.  Also  this  controller  has  four  lights  up  here,  they’re  labelled  A,  B,  C  and  D, 
and  they  show  which  chemicals  are  being  added  to  the  water.  Now  the  setting  of  the 
switches  controls  which  chemicals  are  added,  and  each  chemical  is  represented  by  a 
light,  so  if  I  move  these  switches  around,  you  see  that  different  lights  turn  on  and  off. 

And  because  you  can  get  four  different  settings  of  the  switches,  you  can  also  get  four 
different  mixtures  of  chemicals. 

So  to  begin,  here  is  this  controller  with  two  switches  and  four  lights  and  your  job  is  to 
learn  how  to  operate  it.  First,  let’s  make  sure  that  there  really  are  just  four  ways  to 
set  these  switches.  Each  switch  has  exactly  two  settings,  up  and  down.  So  if  1  is  up, 
then  1  could  be  up  or  2  could  be  down,  and  that’s  two  settings.  Now  Switch  1  could 
be  changed  to  down,  and  then  still  2  could  be  up  or  down.  So  that’s  another  two 
settings,  and  which  makes  four  settings  altogether.  So  this  fertilizer  controller  can 
only  select  among  four  mixtures  of  chemicals  to  add  to  the  water. 

Now  each  of  these  switch  settings  has  its  own  pattern  of  lights,  that  is,  some  of  the 
lights  are  on  and  others  are  off  when  the  switches  are  set  that  way.  So  for  the  first 
setting,  where  both  switches  are  up,  we  have  lights  A,  C  and  D  on.  And  for  the 
second  setting,  where  both  switches  are  down,  lights  A,  B,  and  D  are  on.  For  the 
third  setting,  with  switch  1  up  and  switch  2  down,  we  get  no  lights  on  at  all.  And  for 
the  last  setting,  with  1  down  and  2  up,  we  see  that  only  lights  A  and  C  are  on.  So 
that’s  all  four  cases. 
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Now  this  diagram  summarizes  everything  that  we  said  before.  The  top  indicates  the 
four  lights,  for  the  four  chemicals,  and  the  left  side  shows  the  four  different  switch 
settings.  So  first  both  1  and  2  are  up,  and  that  has  A,  C,  and  D  on.  Next,  both  1 
and  2  are  down,  and  that  has  A  B  and  D  on.  Then  1  is  up  and  2  is  down,  and  that 
has  no  lights  on  at  all.  And  Anally,  1  is  down  and  2  is  up,  and  that  has  A  and  C  on. 
So  this  diagram  shows  us  all  of  this. 

Now  suppose  that  your  supervisor  wants  you  to  get  a  particular  mixture  of  chemicals. 
For  example,  maybe  she  wants  to  get  the  mixture  A,  C,  D.  Then  all  you  have  to  do  is 
look  at  the  diagram  and  see  which  switch  setting  gives  that  mixture.  So,  the  Arst  line 
shows  us  that  both  switches  up  gives  the  mixture  she  wants. 

Another  thing  that  the  supervisor  might  want  is  just  any  setting  that  will  give  the 
chemical  indicated  by  light  D.  Looking  at  the  diagram,  we  can  see  that  if  either 
Switch  1  and  Switch  2  are  both  up,  or  else  if  Switch  1  and  Switch  2  are  both  down, 
we  get  some  of  the  chemical  indicated  by  light  D. 

Final  Summary: 

So  as  you’ve  seen,  the  way  this  controller  works  is  that  the  four  lights  depend  on  the 
settings  of  the  two  switches,  and  the  different  patterns  of  lights  depends  on  the 
settings  of  the  switches.  So  if  you  want  to  get  a  particular  mixture  of  chemicals,  you 
have  to  know  which  switch  settings  give  it. 


n.1.8.  C8  Script 

This  is  the  case  of  Reasoning  without  Visual  Deixis  and  with  Structure  Marking.  The  text  is 
identical  to  that  of  C7,  but  the  actor  does  not  use  Visual  Deixis. 


H.2  Boolean  Level  Stimuli 

B  Series 
A  diagram 
B7  (see  Figure 

III) 

C  =  Sw2 
X  =  "Sw2 
Y  =  Swl  AND  Sv2 
B  =  "Swl  AND  ~Sw2 
Z  =  Svl  oR  sw2 

D  =  (Svl  AND  Sv2)  OR  (~Svl  AND  ~Sv2) 

A  =  (Svl  AND  Sv2)  OR  'Svl 


This  section  gives  the  scripts  used  in  producing  the  experimental  stimuli  for  the 
Experiments.  All  these  stimuli  emply  Reasoning  at  the  Boolean  cognitive  level 
consisting  of  the  equations  mentioned  in  the  text  was  used  for  all  stimuli  except 


Figure  II- 1:  B  Series  Diagram 


II.2.1.  Bl  Script 

This  is  the  basic  case  of  Reasoning  Structure  at  the  Boolean  cognitive  level. 

Initial  Summary: 

Right  now  this  controller  only  gives  certain  particular  mixtures  of  chemicals  and 
water.  But  actually,  it  can  be  programmed,  so  that  the  same  switch  settings  produce 
different  mixtures.  Now  when  you’re  controlling  the  system,  you  may  have  to  figure 
out  which  switch  settings  produce  a  certain  chemical.  Then  it  will  help  a  lot  if  you 
have  a  simple  description  of  when  each  chemical  is  present.  The  easiest  way  to  get 
these  descriptions  is  to  use  some  concepts  from  logic,  like  AND,  OR,  and  NOT.  Now 
what  I’m  going  to  do,  is  describe  how  this  box  works,  and  then  describe  how  you  can 
use  these  concepts  to  figure  out  how  to  get  the  mixture  of  chemicals  that  your 
supervisor  wants. 

So  as  you  can  see,  this  fertilizer  controller  has  two  switches  at  the  bottom,  and 
they’re  numbered  1  and  2.  And  we’ll  see  that  there  are  four  ways  to  set  these  two 
switches.  Also  this  controller  has  four  lights  up  here,  they’re  labelled  A,  B,  C  and  D, 
and  they  show  which  chemicals  are  being  added  to  the  water.  Now  the  setting  of  the 
switches  controls  which  chemicals  are  added,  and  each  chemical  is  represented  by  a 
light,  so  as  I  move  these  switches  around,  you  can  see  that  different  lights  turn  off 
and  on.  And  because  you  can  get  four  different  settings  of  the  switches,  you  can  also 
have  four  different  combinations  of  chemicals. 

You  might  think  you  can  just  memorize  what  happens  for  each  switch  setting.  But 
the  box  is  programmable,  so  even  if  you  memorize  one  way  that  the  box  might  work, 
there  are  a  lot  of  other  ways  that  you  won’t  know  about,  and  there  are  too  many 
possibilities  to  memorize  them  all.  So  it  would  really  help  if  we  could  just  work  with 
simple  descriptions  for  all  the  ways  in  a  particular  program  that  a  given  light  could 
be  on.  This  is  what  we’re  going  to  use  formal  descriptions  for.  Now  we’ll  go  through 
each  of  the  lights,  showing  how  to  describe  which  switch  settings  turn  them  on. 

The  simplest  case  is  light  C,  which  is  on  when  switch  2  is  up,  no  matter  how  switch  1 
is  set,  and  off  when  switch  2  is  down,  no  matter  how  switch  1  is  set.  We  can  express 
this  by  saying  "C  equals  switch  2  V 
C  =  Sw2 

Now  when  we  say  "Sw2"  here,  it  just  means  that  "switch  2  is  on." 

If  some  light,  let’s  call  it  X,  worked  just  the  opposite  from  light  C,  that  is,  if  it  were 
on  when  switch  2  is  down  and  off  when  switch  2  is  up,  no  matter  how  switch  1  is  set, 
we  would  say  "X  equals  NOT  Sw2". 

X  =  ~Sw2 

That  is,  "not"  means  to  take  the  opposite  of  whatever  the  value  is. 

To  describe  more  complicated  relations,  we  also  need  AND.  We  could  have  a  light, 
call  it  Y,  that  is  on  if  both  switch  1  and  switch  2  are  up,  and  is  off  otherwise.  Then 
we  would  say  "Y  equals  switch  1  AND  switch  2". 


Y  =  Swl  AND  Sw2  . 


I 

Now  this  particular  program  for  the  box  doesn’t  have  a  light  like  Y,  but  light  B  is 
only  on  when  both  switches  are  down.  So  we  can  say  *B  equals  NOT  Switch  1  AND 
NOT  Switch  2  V 

B  =  ~Swl  AND  ~Sw2  . 

We  also  need  OR.  In  this  kind  of  formal  description,  OR  is  a  bit  different  from  the 
way  we  use  it  in  English.  If  we  have  a  light  described  by  "Z  equals  Switch  1  OR 
Switch  2", 

Z  =  Swl  OR  Sw2 

then  light  Z  is  on  if  either  switch  1  is  up,  or  if  switch  2  is  up,  or  if  both  switches  are 
up.  And  Z  is  off  in  all  other  cases.  We’ll  use  this  kind  of  OR  when  we  describe  how 
lights  D  and  A  work.  But  first,  notice  that  we  can  use  combinations  of  AND,  OR  and 
NOT  to  form  mote  complicated  descriptions.  For  example,  the  description  that  we 
gave  for  B  combines  two  NOTs  with  an  AND. 

Now  let’s  describe  light  D.  It  is  on  when  both  switches  have  the  same  setting.  This 
means  that  either  both  switches  are  up  OR  both  switches  are  down.  We  can  describe 
this  as  "D  equals  Switch  1  AND  Switch  2  OR  (pause)  NOT  Switch  1  AND  NOT 
Switch  2". 

D  =  (Swl  AND  Sw2)  OR  (~Swl  AND  ~Sw2)  . 

Finally,  let’s  describe  light  A,  which  may  seem  to  be  the  most  difficult  case.  A  is  on 
if  both  switches  are  up,  or  if  switch  1  is  down,  no  matter  where  2  is,  and  is  off  in  all 
other  cases.  We  can  describe  this  as  "A  equals  Switch  1  AND  Switch  2  OR  (pause) 

NOT  Switch  1*. 

A=  (Swl  AND  Sw2)  OR  ~Swl  . 

Now  suppose  that  your  supervisor  wants  you  to  get  the  chemical  indicated  by  light 
B.  Looking  at  the  description  for  B,  we  can  see  that  B  is  on  only  if  both  Switch  1  and 
Switch  2  are  both  down.  (But  if  she  asks  for  some  other  chemical,  there  may  be  more 
than  one  setting  of  switches  that  will  work.) 

Another  thing  your  supervisor  might  want  is  to  get  a  particular  mixture  of  chemicals. 

For  example,  she  might  want  to  get  the  mixture  A,  B,  D.  Now  the  description  for  A 
will  tell  you  which  switch  settings  have  A  on,  and  then  you  can  look  at  the 
description  for  B  to  see  which  settings  for  A  also  have  B  on,  and  finally  use  the 
description  for  D  to  see  which  of  these  have  D  on  too.  Actually,  it  would  be  easier  to 
start  with  B,  since  we  now  know  there  is  only  one  setting  where  B  is  on,  and  then  you 
can  just  see  if  A  and  D  are  also  on  when  Switch  1  and  Switch  2  are  both  up. 

Final  Summary: 

So  as  you’ve  seen,  the  way  this  controller  works  is  that  the  four  lights  indicate  the 
chemicals  and  the  different  patterns  of  lights  depend  on  the  settings  of  the  two 


switches.  So  if  you  want  to  get  a  particular  mixture  of  chemicals,  you  have  to  know 
which  switch  setting  gives  that  mixture.  And  if  you  want  to  know  how  to  get  a 
particular  chemical,  you  have  to  know  which  switch  settings  give  it.  And  an  easy 
way  to  express  the  various  possible  switch  settings  is  to  use  these  concepts  of  AND, 

OR,  and  NOT  from  logic. 

n.2.2.  B2  Script 

This  is  the  case  of  Boolean  level  Reasoning,  without  an  initial  summary.  The  text  is  identical 

to  that  of  Bl,  except  that  the  initial  summary  is  replaced  by  the  following  filler  proposition: 
So,  as  you  can  see,  we  have  here  an  irrigation  controller,  which  controls  the  process  of 
adding  chemicals  to  the  water  that  goes  out  to  irrigate  the  different  crops  in  various 
fields.  Different  kinds  of  crops  need  different  mixtures  of  fertilizers  to  help  them 
reach  their  optimum  growth,  so  the  operator  of  the  controller  has  to  be  able  to 
change  the  mixture  of  chemicals,  depending  on  what  crops  are  being  grown,  what  the 
temperature  is,  how  much  sunlight  there  is,  how  much  rain  there  is,  and  so  on.  And 
the  operator  of  the  controller  device  would  make  the  changes  when  he  or  she  received 
orders  by  telephone  from  a  supervisor  who  had  more  information  about  the  whole 
system. 

The  length  of  this  is  approximately  equal  to  that  of  the  original  initial  summary. 

n.2.3.  B3  Script 

This  is  the  case  of  Reasoning  Structure  without  a  final  summary.  The  text  is  identical  to  that 

of  Bl,  except  that  the  final  summary  is  replaced  by  the  following  filler  proposition: 

So  if  you  were  to  get  this  job  working  as  the  operator  of  an  irriguation  control 
system,  that’s  what  your  job  would  be.  Of  course,  we’ve  gone  over  only  part  of  the 
job.  You  would  also  be  monitoring  other  devices  as  well,  checking  that  many  other 
irrigation  systems  were  functioning  correctly,  that  there  were  no  blockages  in  the 
lines,  or  breaks  or  leaks  in  the  system,  and  that  the  chemicals  were  flowing  correctly, 
and  so  on. 

The  length  of  this  is  approximately  equal  to  that  of  the  original  final  summary. 

n.2.4.  B4  Script 

This  is  the  case  Boolean  level  Reasoning  with  sentential  focus  inconsistencies. 

Initial  Summary: 

Right  now  this  controller  only  gives  certain  particular  mixtures  of  chemicals  and 
water.  But  actually,  it  can  be  programmed,  so  that  the  same  switch  settings  produce 
different  mixtures.  Now,  you  may  have  to  figure  out  how  to  produce  a  certain 
chemical  when  you’re  controlling  the  system.  Then  it  will  help  a  lot  if  you  have  a 
simple  description  of  which  switch  settings  produce  that  chemical.  The  easiest  way  to 
get  these  descriptions  is  to  use  some  concepts  from  logic,  like  AND,  OR,  and  NOT. 


Now  what  I’m  going  to  do,  is  describe  how  this  box  works,  and  then  describe  how  you 
can  use  these  concepts  to  get  the  mixture  of  chemicals  that  your  supervisor  wants. 

So  as  you  can  see,  this  fertilizer  controller  has  two  switches  at  the  bottom,  and  also 
four  lights  up  here.  Now  the  switches  are  numbered  1  and  2,  and  the  lights  are 
labelled  A,  B,  C  and  D.  And  we  will  see  that  there  are  four  ways  to  set  the  two 
switches.  Now  the  lights  show  which  chemicals  are  being  added  to  the  water.  And 
the  setting  of  the  switches  controls  which  chemicals  are  added.  Now  each  chemical  is 
represented  by  a  light.  So,  as  I  move  these  switches  around,  you  can  see  that 
different  lights  turn  off  and  on.  And  so  you  can  get  four  different  combinations  of 
chemicals,  because  there  are  just  four  different  settings  of  the  switches. 

You  might  think  you  can  just  memorize  what  happens  for  each  switch  setting.  But 
the  box  is  programmable,  so  even  if  you  memorize  one  way  that  the  box  might  work, 
there  are  a  lot  of  other  ways  that  you  won’t  know  about,  and  there  are  too  many 
possibilities  to  memorize  them  all.  So  it  would  really  help  if  we  could  just  work  with 
simple  descriptions  for  all  the  ways  in  a  particular  program  that  a  given  light  could 
be  on.  This  is  what  we’re  going  to  use  formal  descriptions  for.  Now  we’ll  go  through 
the  different  cases  and  show  how  different  switch  settings  give  different  lights. 

The  simplest  case  is  light  C.  Now  when  switch  2  is  up,  then  light  C  is  on,  no  matter 
how  switch  1  is  set.  And  light  C  is  off  when  switch  2  is  down,  no  matter  how  switch 
1  is  set.  We  can  express  this  by  saying  "C  equals  switch  2", 

C  =  Sw2 

Now  when  we  say  "Sw2"  here,  it  just  means  that  "switch  2  is  on." 

We  might  have  a  light,  let’s  call  it  X,  that  worked  just  the  opposite  from  light 
C.  That  is,  when  switch  2  is  down  X  would  be  on,  and  it  would  be  off  when  switch  2 
is  up,  no  matter  how  switch  1  is  set.  Then  we  would  say  "X  equals  NOT  Sw2". 

X  =  ~Sw2 

That  is,  "not"  means  to  take  the  opposite  of  whatever  the  value  is. 

To  describe  more  complicated  relations,  we  also  need  AND.  So  let’s  imagine  another 
light,  call  it  Y,  and  say  that  if  both  switch  1  and  switch  2  are  up,  then  Y  is  on,  and 
otherwise  it  is  off.  Then  we  would  say  "Y  equals  switch  1  AND  switch  2". 

Y  =  Swl  AND  Sw2  . 

Now  this  particular  program  for  the  box  doesn’t  have  a  light  like  Y,  but  only  when 
both  switches  are  down  can  we  have  light  B  on.  So  we  can  say  "B  equals  NOT 
Switch  1  AND  NOT  Switch  2". 

B  =  ~SwI  AND  ~Sw2  . 

We  also  need  OR.  In  this  kind  of  formal  description,  OR  is  a  bit  different  from  the 
way  we  use  it  in  English.  We  might  have  a  light  described  by  "Z  equals  Switch  I  OR 
Switch  2", 

Z  =  Swl  OR  Sw2  . 


So  if  either  switch  1  is  up,  or  if  switch  2  is  up,  or  if  both  switches  are  up,  then  light  Z 
is  on.  And  Z  is  off  in  all  other  cases.  We’ll  use  this  kind  of  OR  when  we  describe 
how  lights  D  and  A  work.  But  first,  notice  that  we  can  use  combinations  of  AND, 

OR  and  NOT  to  form  more  complicated  descriptions.  For  example,  the  description 
that  we  gave  for  B  combines  two  NOTs  with  an  AND. 

Now  let’s  describe  light  D.  When  both  switches  have  the  same  setting,  then  D  is  on. 
This  means  D  is  on  if  either  both  switches  are  up  OR  both  switches  are  down.  We 
can  describe  this  as  "D  equals  Switch  1  AND  Switch  2  OR  (pause)  NOT  Switch  1 
AND  NOT  Switch  2*. 

D  =  (Swl  AND  Sw2)  OR  (~Swl  AND  ~Sw2)  . 

Finally,  let’s  describe  light  A,  which  may  seem  to  be  the  most  difficult  case.  If  both 
switches  are  up  then  A  is  on,  OR  it  is  on  if  switch  1  is  down,  no  matter  where  2  is, 
and  in  all  other  cases  A  is  off.  We  can  describe  this  as  "A  equals  Switch  1  AND 
Switch  2  OR  (pause)  NOT  Switch  1". 

A=  (Swl  AND  Sw2)  OR  ~Swl  . 

Now  suppose  that  your  supervisor  wants  you  to  get  the  chemical  indicated  by  light 
B.  We  can  see  that  only  both  Switch  1  and  Switch  2  down  gives  light  B  on,  from  the 
description  of  B.  (But  if  she  asks  for  some  other  chemical,  there  may  be  more  than 
one  setting  of  switches  that  will  work.) 

Another  thing  your  supervisor  might  want  is  to  get  the  switch  settings  for  a  particular 
mixture  of  chemicals.  For  example,  she  might  want  to  get  the  mixture  A,  B,  D.  Now 
you  can  find  which  switch  settings  have  A  on  from  the  description  of  A,  and  then  you 
can  look  at  the  description  of  B  to  see  which  settings  for  A  also  have  B  on,  and  finally 
you  can  get  the  switch  settings  for  D  from  its  description  and  see  which  of  these  have 
A  and  B  on  too.  Actually,  it  would  be  easier  to  start  with  B,  since  we  now  know 
there  is  only  one  setting  where  B  is  on,  and  then  you  can  just  see  if  A  and  D  are  also 
on  when  Switch  1  and  Switch  2  are  both  up. 

Final  Summary: 

So  as  you’ve  seen,  the  way  this  controller  works  is  that  the  four  lights  indicate  the 
chemicals,  and  the  different  settings  of  the  two  switches  produce  different  patterns  of 
lights.  So  if  you  want  to  get  a  particular  mixture  of  chemicals,  you  have  to  know 
which  switch  setting  gives  that  mixture.  And  if  you  know  which  switch  settings  give 
a  particular  chemical,  you  can  get  that  chemical  with  them.  So  an  easy  way  to 
express  the  various  possible  switch  settings  is  to  use  these  concepts  of  AND,  OR,  and 
NOT  from  logic. 
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n.2.5.  B5  Script 

This  is  the  case  of  Boolean  level  Reasoning  with  discourse  subunit  boundary  violations. 
Initial  Summary: 

Right  now  this  controller  only  gives  certain  particular  mixtures  of  chemicals  and 
water.  And  when  you’re  controlling  the  system,  you  may  have  to  figure  out  which 
switch  settings  produce  a  certain  chemical.  Then  it  will  help  a  lot  if  you  have  a 
simple  description  of  when  each  chemical  is  present.  The  easiest  way  to  get  these 
descriptions  is  to  use  some  concepts  from  logic,  like  AND,  OR,  and  NOT.  This  is 
because  the  controller  can  be  programmed,  so  that  the  same  switch  settings  produce 
different  mixtures  in  other  programs. 

So  as  you  can  see,  this  fertilizer  controller  has  two  switches  at  the  bottom,  and 
they’re  numbered  1  and  2.  Now  what  I’m  going  to  do,  is  describe  how  this  box 
works,  and  then  describe  how  you  can  use  these  concepts  from  logic  to  figure  out  how 
to  get  the  mixture  of  chemicals  that  your  supervisor  wants.  And  we’ll  see  that  there 
are  four  ways  to  set  these  two  switches.  Now  the  setting  of  the  switches  controls 
which  chemicals  are  added,  and  each  chemical  is  represented  by  a  light,  so  as  I  move 
these  switches  around,  you  can  see  that  different  lights  turn  off  and  on.  Also  this 
controller  has  four  lights  up  here,  they’re  labelled  A,  B,  C  and  D,  and  they  show 
which  chemicals  are  being  added  to  the  water. 

Now  because  you  can  get  four  different  settings  of  the  switches,  you  can  also  have 
four  different  combinations  of  chemicals.  You  might  think  you  can  just  memorize 
what  happens  for  each  switch  setting.  But  the  box  is  programmable,  so  even  if  you 
memorize  one  way  that  the  box  might  work,  there  are  a  lot  of  other  ways  that  you 
won’t  know  about.  And  because  there  are  too  many  possibilities  to  memorize  them 
all,  it  would  really  help  if  we  could  just  work  with  simple  descriptions  for  all  the  ways 
in  a  particular  program  that  a  given  light  could  be  on.  This  is  what  we’re  going  to 
use  formal  descriptions  for. 

The  simplest  case  is  light  C.  What  we’re  doing  now  is  going  through  each  of  the 
lights,  showing  how  to  describe  which  switch  settings  turn  them  on.  So  C  is  on  when 
switch  2  is  up,  no  matter  how  switch  1  is  set,  and  off  when  switch  2  is  down,  no 
matter  how  switch  I  is  set.  Now  when  we  say  "Sw2,"  we  just  mean  that  "switch  2  is 
on",  then  we  can  describe  when  light  C  is  on  by  saying  "C  equals  switch  2". 

C  =  Sw2  . 

Imagine  a  light,  let’s  call  it  X,  that  is  on  when  switch  2  is  down  and  off  when  switch  2 
is  up,  no  matter  how  switch  1  is  set.  Then  X  works  just  the  opposite  from  light 
C.  And  if  we  let  "NOT"  mean  to  take  the  opposite  of  whatever  the  value  is,  then  we 
can  say  "X  equals  NOT  Sw2". 

X  =  ~Sw2  . 

We  could  also  have  a  light,  call  it  Y,  that  is  on  if  both  switch  1  and  switch  2  are  up, 


and  is  off  otherwise.  We  need  AND  to  describe  relations  like  this.  For  example,  we 
would  say  “Y  equals  switch  1  AND  switch  2". 

Y  =  Swl  AND  Sw2  . 

Now  light  B  is  only  on  when  both  switches  are  down,  so  it  is  only  a  little  different 
from  light  Y.  It  can  be  described  by  saying  "B  equals  NOT  Switch  1  AND  NOT 
Switch  2". 

B  =  ~Swl  AND  ~Sw2  . 

Also  imagine  a  light  described  by  "Z  equals  Switch  1  OR  Switch  2". 

Z  =  Swl  OR  Sw2  . 

In  this  kind  of  formal  description,  OR  is  a  bit  different  from  the  way  we  use  it  in 
English,  since  light  Z  is  on  if  either  switch  1  is  up,  or  if  switch  2  is  up,  or  if  both 
switches  are  up.  And  Z  is  off  in  all  other  cases. 

Notice  that  we  can  use  combinations  of  AND,  OR  and  NOT  to  form  more 
complicated  descriptions.  For  example,  the  description  that  we  gave  for  B  combines 
two  NOTs  with  an  AND.  We’ll  also  use  this  kind  of  OR  when  we  describe  how  lights 
D  and  A  work. 

Now  let's  describe  light  D.  It  is  on  when  both  switches  have  the  same  setting.  This 
means  that  either  both  switches  are  up  OR  both  switches  are  down.  We  can  describe 
this  as  "D  equals  Switch  1  AND  Switch  2  OR  (pause)  NOT  Switch  1  AND  NOT 
Switch  2V 

D  =  (Swl  AND  Sw2)  OR  (~Swl  AND  ~Sw2)  . 

Finally,  let’s  describe  light  A.  It  is  on  if  both  switches  are  up,  or  if  switch  1  is  down, 
no  matter  where  2  is,  and  is  off  in  all  other  cases.  We  can  describe  this  as  "A  equals 
Switch  1  AND  Switch  2  OR  (pause)  NOT  Switch  1". 

A=  (Swl  AND  Sw2)  OR  ~Swl  . 

This  may  seem  to  be  the  most  difficult  case. 

Now  suppose  that  your  supervisor  wants  you  to  get  the  chemical  indicated  by  light 
B.  If  she  asks  for  some  other  chemical,  there  may  be  more  than  one  setting  of 
switches  that  will  work,  but  looking  at  the  description  for  B,  we  can  see  that  B  is  on 
only  if  Switch  1  and  Switch  2  are  both  down. 

Another  thing  your  supervisor  might  want  is  to  get  a  particular  mixture  of  chemicals. 
Then  you  can  use  the  descriptions  to  see  which  settings  work.  Thus,  the  description 
for  A  will  tell  you  which  switch  settings  have  A  on.  Suppose  your  supervisor  wants  to 
get  the  mixture  A,  B,  D.  Then  you  can  look  at  the  description  for  B  to  see  which 
settings  for  A  also  have  B  on,  and  finally  use  the  description  for  D  to  see  which  of 
these  have  D  on  too.  Actually,  it  would  be  easier  to  start  with  B,  since  we  now  know 
there  is  only  one  setting  where  B  is  on,  and  then  you  can  just  see  if  A  and  D  are  also 
on  when  Switch  1  and  Switch  2  are  both  down. 


Final  Summary: 

So  as  you’ve  seen,  the  way  this  controller  works  is  that  the  four  lights  indicate  the 
chemicals.  So  if  you  want  to  get  a  particular  mixture  of  chemicals,  you  have  to  know 
which  switch  setting  gives  that  mixture,  because  the  different  patterns  of  lights 
depend  on  the  settings  of  the  two  switches.  And  if  you  want  to  know  how  to  get  a 
particular  chemical,  you  have  to  know  which  switch  settings  give  it.  And  an  easy 
way  to  express  the  various  possible  switch  settings  is  to  use  these  concepts  of  AND, 
OR,  and  NOT  from  logic. 

n.2.6.  B6  Script 

This  is  the  case  of  Boolean  level  Reasoning  with  discourse  subunits  in  a  scrambled  order. 

So  as  you  can  see,  this  fertilizer  controller  has  two  switches  at  the  bottom,  and 
they’re  numbered  1  and  2.  And  we’ll  see  that  there  are  four  ways  to  set  these  two 
switches.  Also  this  controller  has  four  lights  up  here,  they’re  labelled  A,  B,  C  and  D, 
and  they  show  which  chemicals  are  being  added  to  the  water.  Now  the  setting  of  the 
switches  controls  which  chemicals  are  added,  and  each  chemical  is  represented  by  a 
light,  so  as  I  move  these  switches  around,  you  can  see  that  different  lights  turn  off 
and  on.  And  because  you  can  get  four  different  settings  of  the  switches,  you  can  also 
have  four  different  combinations  of  chemicals. 

The  simplest  case  is  light  C,  which  is  on  when  switch  2  is  up,  no  matter  how  switch  1 
is  set,  and  off  when  switch  2  is  down,  no  matter  how  switch  1  is  set.  We  can  express 
this  by  saying  "C  equals  switch  2*. 

C  =  Sw2 

Now  when  we  say  "Sw2"  here,  it  just  means  that  "switch  2  is  on." 

If  some  light,  let’s  call  it  X,  worked  just  the  opposite  from  light  C,  that  is,  if  it  were 
on  when  switch  2  is  down  and  off  when  switch  2  is  up,  no  matter  how  switch  1  is  set, 
we  would  say  "X  equals  NOT  Sw2". 

X  —  ~Sw2 

That  is,  "not"  means  to  take  the  opposite  of  whatever  the  value  is. 

Initial  Summary: 

Right  now  this  controller  only  gives  certain  particular  mixtures  of  chemicals  and 
water.  But  actually,  it  can  be  programmed,  so  that  the  same  switch  settings  produce 
different  mixtures.  Now  when  you’re  controlling  the  system,  you  may  have  to  figure 
out  which  switch  settings  produce  a  certain  chemical.  Then  it  will  help  a  lot  if  you 
have  a  simple  description  of  when  each  chemical  is  present.  The  easiest  way  to  get 
these  descriptions  is  to  use  some  concepts  from  logic,  like  AND,  OR,  and  NOT.  Now 
what  I’m  going  to  do,  is  describe  how  this  box  works,  and  then  describe  how  you  can 
use  these  concepts  to  figure  out  how  to  get  the  mixture  of  chemicals  that  your 
supervisor  wants. 

To  describe  more  complicated  relations,  we  also  need  AND.  We  could  have  a  light, 


call  it  Y,  that  is  on  if  both  switch  1  and  switch  2  are  up,  and  is  off  otherwise.  Then 
we  would  say  "Y  equals  switch  1  AND  switch  2". 

Y  =  Swl  AND  Sw2  . 

Now  this  particular  program  for  the  box  doesn’t  have  a  light  like  Y,  but  light  B  is 
only  on  when  both  switches  are  down.  So  we  can  say  "B  equals  NOT  Switch  1  AND 
NOT  Switch  2". 

B  =  ~Swl  AND  ~Sw2  . 

We  also  need  OR.  In  this  kind  of  formal  description,  OR  is  a  bit  different  from  the 
way  we  use  it  in  English.  If  we  have  a  light  described  by  "Z  equals  Switch  1  OR 
Switch  2*, 

Z  =  Swl  OR  Sw2 

then  light  Z  is  on  if  either  switch  1  is  up,  or  if  switch  2  is  up,  or  if  both  switches  are 
up.  And  Z  is  off  in  all  other  cases.  We’ll  use  this  kind  of  OR  when  we  describe  how 
lights  D  and  A  work.  But  first,  notice  that  we  can  use  combinations  of  AND,  OR  and 
NOT  to  form  more  complicated  descriptions.  For  example,  the  description  that  we 
gave  for  B  combines  two  NOTs  with  an  AND. 

You  might  think  you  can  just  memorize  what  happens  for  each  switch  setting.  But 
the  box  is  programmable,  so  even  if  you  memorize  one  way  that  the  box  might  work, 
there  are  a  lot  of  other  ways  that  you  won’t  know  about,  and  there  are  too  many 
possibilities  to  memorize  them  all.  So  it  would  really  help  if  we  could  just  work  with 
simple  descriptions  for  all  the  ways  in  a  particular  program  that  a  given  light  could 
be  on.  This  is  what  we’re  going  to  use  formal  descriptions  for.  Now  we’ll  go  through 
each  of  the  lights,  showing  how  to  describe  which  switch  settings  turn  them  on. 

Final  Summary: 

So  as  you’ve  seen,  the  way  this  controller  works  is  that  the  four  lights  indicate  the 
chemicals  and  the  different  patterns  of  lights  depend  on  the  settings  of  the  two 
switches.  So  if  you  want  to  get  a  particular  mixture  of  chemicals,  you  have  to  know 
which  switch  setting  gives  that  mixture.  And  if  you  want  to  know  how  to  get  a 
particular  chemical,  you  have  to  know  which  switch  settings  give  it.  And  an  easy 
way  to  express  the  various  possible  switch  settings  is  to  use  these  concepts  of  AND, 
OR,  and  NOT  from  logic. 

Now  suppose  that  your  supervisor  wants  you  to  get  the  chemical  indicated  by  light 
B.  Looking  at  the  description  for  B,  we  can  see  that  B  is  on  only  if  Switch  1  and 
Switch  2  are  both  down.  (But  if  she  asks  for  some  other  chemical,  there  may  be  more 
than  one  setting  of  switches  that  will  work.) 

Another  thing  your  supervisor  might  want  is  to  get  a  particular  mixture  of  chemicals. 
For  example,  she  might  want  to  get  the  mixture  A,  B,  D.  Now  the  description  for  A 
will  tell  you  which  switch  settings  have  A  on,  and  then  you  can  look  at  the 
description  for  B  to  see  which  settings  for  A  also  have  B  on,  and  finally  use  the 


53 


description  for  D  to  see  which  of  these  have  D  on  too.  Actually,  it  would  be  easier  to 
start  with  6,  since  we  now  know  there  is  only  one  setting  where  B  is  on,  and  then  you 
can  just  see  if  A  and  D  are  also  on  when  Switch  1  and  Switch  2  are  both  up. 

Now  let’s  describe  light  D.  It  is  on  when  both  switches  have  the  same  setting.  This 
means  that  either  both  switches  are  up  OR  both  switches  are  down.  We  can  describe 
this  as  "D  equals  Switch  1  AND  Switch  2  OR  (pause)  NOT  Switch  1  AND  NOT 
Switch  2“. 

D  =  (Swl  AND  Sw2)  OR  (~Swl  AND  ~Sw2) . 

Finally,  let’s  describe  light  A,  which  may  seem  to  be  the  most  difficult  case.  A  is  on 
if  both  switches  are  up,  or  if  switch  1  is  down,  no  matter  where  2  is,  and  is  off  in  all 
other  cases.  We  can  describe  this  as  "A  equals  Switch  1  AND  Switch  2  OR  (pause) 
NOT  Switch  1-. 

A=  (Swl  AND  Sw2)  OR  ~Swl  . 


II.2.7.  B7  Script 

This  is  the  case  of  Boolean  level  Reasoning  with  a  diagram  that  is  not  coordinated  with  the 
text.  The  text  is  identical  to  that  of  Bl,  but  Figure  12  is  used  instead  of  Figure  II- 1. 


IQ.  Test  Questions 

This  appendix  contains  the  questionaires  whose  scores  are  used  as  dependent  variables  for  the  C 
and  B  Series  of  stimuli. 

m.l  Combinatorial  Series  Questions 

1.  How  many  mixtures  that  contain  chemical  A  can  you  get  with  this  controller? 

2.  Which  switch  settings  produce  pure  water?  (Check  all  the  applicable  switch  settings.) 

3.  Can  you  get  fertilized  water  consisting  of  the  mixture  A,  C,  D  of  chemicals? 

4.  Can  you  get  fertilized  water  consisting  of  exactly  the  mixture  A,  D  of  chemicals? 

5.  How  many  different  switch  settings  yield  either  chemical  B  or  else  the  chemicals  A  and  C? 

6.  Does  any  switch  setting  produce  water  containing  all  the  chemicals? 

7.  How  many  mixtures  that  contain  both  chemicals  A  and  C  can  you  get  with  this 
controller? 

8.  How  many  different  mixtures  of  fertilized  or  unfertilized  water  does  this  controller 
produce? 

9.  How  many  different  switch  settings  contain  either  chemical  D  or  else  the  chemicals  A  and 
C? 

10.  Do  previous  switch  settings  have  any  effect  on  the  result  of  a  switch  setting? 

11.  Do  the  following  describe  the  same  switch  settings? 

a.  Swl  and  Sw2  are  the  same. 

b.  Swl  and  Sw2  are  both  up  or  else  Swl  and  Sw2  are  both  down. 

12.  Do  the  following  describe  the  same  switch  settings? 

a.  Either  Swl  is  up  or  else  both  Swl  and  Sw2  are  down. 

b.  Either  Sw2  is  down  or  Swl  is  up. 

13.  Do  the  following  describe  the  same  switch  settings? 

a.  Either  Sw2  is  up  or  else  Swl  is  down. 

b.  Either  Swl  and  Sw2  are  up  or  else  Swl  is  down. 

(Question  5  was  not  used  in  computing  C  Series  scores,  since  its  entry  on  the  answer  sheet  was 
flawed.) 

HI. 2  Boolean  Series  Questions 

1.  How  many  mixtures  that  contain  chemical  A  can  you  get  with  this  controller? 

2.  Which  switch  settings  produce  pure  water?  (Check  all  the  applicable  switch  settings.) 

3.  Can  you  get  fertilized  water  consisting  of  exactly  the  mixture  A,  C,  D  of  chemicals? 


4.  Can  you  get  fertilized  water  consisting  of  exactly  the  mixture  A,  D  of  chemicals? 

5.  How  many  different  switch  settings  yield  either  chemical  B  or  else  the  chemicals  A  and  C? 

6.  Does  any  switch  setting  produce  water  containing  all  the  chemicals? 

7.  How  many  mixtures  that  contain  both  chemicals  A  and  C  can  you  get  with  this 
controller? 

8.  What  is  the  maximum  number  of  different  kinds  of  fertilized  water  that  the  controller  can 
produce? 

0.  Do  previous  switch  settings  have  any  effect  on  the  result  of  a  switch  setting? 

10.  Is  it  necessary  to  always  have  one  switch  setting  that  produces  pure  water? 

11.  Do  1  and  2  describe  the  same  switch  settings? 

a.  Swl  AND  Sw2  . 

b.  ~Swl  OR  ~Sw2  . 

12.  Do  1  and  2  describe  the  same  switch  settings? 

a.  Swl  OR  (~Swl  AND  ~Sw2) . 

b.  ~Sw2  OR  Swl  . 

13.  Do  1  and  2  describe  the  same  switch  settings? 

a.  ~(Swl  OR  Sw2)  . 

b.  ~Swl  AND  ~Sw2  . 

14.  Do  1  and  2  describe  the  same  switch  settings? 

a.  Swl  AND  (Swl  OR  Sw2)  . 

b.  Swl  AND  Sw2  . 

15.  Do  1  and  2  describe  the  same  switch  settings? 

a.  Swl  AND  ~Sw2  . 

b.  ~Swl  OR  Sw2  . 

16.  Do  1  and  2  describe  the  same  switch  settings? 

a.  — Swl  AND  ~Sw2  . 

b.  Swl  OR  Sw2  . 
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IV.  Papers  in  Preparation 


Three  papers  are  in  preparation  reflecting  results  produced  under  sponsorship  of  this  project: 

1.  "The  Effect  of  Discourse  Type  on  Comprehension",  by  Joseph  Goguen  and  Charlotte 
Linde.  This  paper  will  present  our  findings  on  the  C  Series  experiments. 

2.  "Visual  Deixis",  by  Charlotte  Linde  and  Joseph  Goguen.  This  paper  will  present  our 
results  on  visual  deixis. 

3.  "Foundations  of  Semiotics,"  by  Joseph  A.  Goguen,  Tora  Bikson  and  Charlotte  Linde. 
This  paper  will  present  our  mathematical  theory  of  semiotic  structure,  emphasizing  its 
applications  to  multi-media  instruction. 
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